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PREFACE 


This  technical  report  contains  the  finding  of  the  3D  Graphics  Investigation  Study  project. 
This  project  is  Delivery  Order  16  of  the  Logistics  Technology  Research  Support  (LTRS) 
program  (Contract  F41624-97-D-5002).  The  work  described  in  this  report  was  performed  during 
the  period  18  December  1997  through  18  August  1998.  The  principal  investigators  for  this  effort 
were  IstLt.  Maurice  Azar  from  AFRL/HESR,  Capt.  Mike  Clark  from  AFRL/HESR,  Nicholas 
Stute  from  TASC,  Inc.,  Holly  Bautsch  and  Matthew  Gdowski  from  Logicon  Technical  Services, 
Inc. 
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INTRODUCTION 


Background 

The  current  trend  within  the  United  States  Air  Force  (USAF)  is  to  replace  the  traditional 
paper-based  aircraft  maintenance  Technical  Orders  (TOs)  with  electronic  TOs.  The  USAF  uses  a 
great  deal  of  photographs  and  graphics  in  TOs,  but  the  graphic  images  are  usually  depicted  as 
fixed  oblique  perspective  views.  Therefore,  paper-based  technical  data  leaves  the  maintainers 
with  no  recourse  or  option  to  manipulate  the  graphics  to  render  a  more  useful  perspective. 
Whether  the  problem  is  one  of  reversed  perspective  (inside-out  image  -  outside-in  real  view), 
mirror  imaging,  or  displaced  orientation,  paper-based  TOs  can  only  provide  a  finite  number  of 
views  in  the  limited  amount  of  space  available.  Maintainers  may  not  intuitively  interpret  these 
perspectives  and  an  alternative  view  may  be  more  useful.  Initial  applications  of  electronic  TOs 
have  used  two-dimensional  graphics  similar  to  those  used  in  the  paper  TOs.  The  computer 
technology  used  to  present  electronic  TOs  provides  a  potential  solution  to  the  problem  of  fixed 
perspective  graphics  -  the  use  of  dynamic  3D  graphics. 

In  today’s  world  of  increasingly  faster  computers  and  increasingly  complex  graphic 
engines,  the  old  method  of  representing  3D  objects  by  orthographic  projection  (three  separate 
static  views  at  90°  rotations)  and  isometric  views  has  given  way  to  perspective  projections  of 
wireframe  and  solid  models.  It  has  long  been  known  that  2D  representations  of  3D  objects  are 
perceived  as  3D  objects  (Shepard  &  Metzler,  1971),  however,  the  question  remains  whether 
computer  display-based  representations  (3D  “models”)  have  a  significant  advantage  over 
standard  printed  graphics.  Unlike  3D  computer  models,  paper-based  graphics  are  not  only  two- 
dimensional,  they  are  further  limited  in  their  ability  to  represent  3D  images  because  of  the  static 
nature  of  the  medium.  The  ability  of  computer-based  displays  to  represent  information 
dynamically  and  allow  interaction  with  3D  models  adds  a  significant  dimension  in  maximizing 
information  transfer  -  3D  model  images  are  not  only  subjectively  preferred,  but  they  have  been 
found  to  increase  performance  as  well  (Bemis,  Leeds  &  Winer,  1988).  The  application  of  this  3D 
graphic  technology  to  electronic  TOs  has  the  potential  to  enhance  the  aircraft  maintainer’s 
efficiency  and  accuracy  when  executing  routine  and  Aircraft  Battle  Damage  Assessment  and 
Repair  (ABDAR)  maintenance  tasks. 
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The  Logistics  Readiness  Branch  (AFRL/HESR)  of  the  Crew  Survivability  and  Logistics 
Division  (AFRL/HES)  conducted  this  study  to  investigate  the  feasibility  of  using  3D  graphic 
technology  in  electronic  TOs.  Part  of  HESR’s  mission  is  to  perform  logistics  technology  research 
and  development  focused  on  improving  the  performance  of  integrated  systems  of  people, 
information,  and  equipment  doing  essential  acquisition  and  logistics  support  functions  in 
peacetime  and  war. 

Current  Study  Objectives  and  Approach 

Research  and  development  in  computer  3D  graphics  has  resulted  in  the  availability  of 
several  hundred  3D  file  formats  and  software  tools  for  designing  and  presenting  3D  graphics. 
The  purpose  of  this  study  was  to  evaluate  3D  graphics  technologies  and  presentation  formats  for 
potential  application  in  electronic  TOs  to  support  complex  maintenance  tasks  such  as  evaluating 
structural  damage  to  aircraft. 

Three  main  objectives  were  identified  for  this  study: 

(1)  Establish  requirements  for  3D  graphics  and  determine  how  aircraft 
maintainers  could  use  them  to  execute  their  job  responsibilities  more  efficiently  and 
accurately. 

(2)  Identify  all  3D  file  formats  and  3D  viewers  currently  used  by  the  commercial 
sector  and  the  Department  of  Defense  (DoD)  and  evaluate  the  identified  3D  file 
formats  and  viewers  to  determine  the  two  “best  value”  file  formats  and  viewers  for 
maintenance  applications. 

(3)  Conduct  a  comparative  study  of  the  two  “best  value”  viewers  using  a  realistic 
maintenance  scenario  to  gather  data  about  the  effectiveness,  usability,  and  limitations 
of  the  “best  value”  file  formats  and  viewers  for  maintenance  tasks. 

The  first  phase  of  the  study  focused  on  performing  a  literature  review  and  conducting  a 
Cognitive  Task  Analysis  (CTA).  The  literature  review  concentrated  on  human  factors  issues  such 
as  user  perception  and  interpretation  of  3D  graphics,  human  factors  aspects  of  the  graphical  user 
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interface  (GUI),  display  thresholds  and  limitations,  interactive  control  of  3D  objects,  screen 
attributes,  and  the  use  of  color.  The  results  of  the  literature  review  can  be  found  in  the  section 
titled  "Literature  Review  of  3D  Graphics  Human  Factor  Issues". 

The  purpose  of  the  CTA  was  to  gather  information  from  aircraft  maintainers  on  their 
views  about  potential  uses  of  3D  graphics  in  a  maintenance  environment.  Background 
information,  including  the  benefits  of  CTA,  is  available  in  the  section  titled  "Cognitive  Task 
Analysis  Background".  A  description  of  the  CTA  conducted  for  this  study  and  its  results  are 
given  in  the  section  titled  "Cognitive  Task  Analysis". 

The  second  phase  of  the  study  concentrated  on  identifying  industry  standard  3D  file 
formats  and  viewers.  The  study  targeted  both  commercially  available  and  DoD  standard  formats 
and  viewers.  The  scope  of  this  effort  was  to  look  exclusively  at  existing  file  formats  and  3D 
viewers  and  not  to  develop  custom  software.  After  identifying  all  available  formats,  the  team 
evaluated  and  ranked  each  of  the  file  formats  and  viewers.  The  first  step  in  the  process  involved 
developing  a  set  of  evaluation  criteria  for  both  the  file  formats  and  viewers.  Once  the  criteria 
were  outlined,  each  of  the  file  formats  and  viewers  were  reviewed  and  ranked.  Based  on  the 
results  of  the  ranking,  two  file  formats  and  two  viewers  were  recommended  as  “best  value”  for 
this  effort.  The  sections  titled  "3D  File  Formats"  and  "3D  Viewers"  contain  the  details  of  this 
phase  of  the  study. 

The  third  and  final  phase  of  the  effort  was  to  conduct  a  comparative  study  of  the  two 
identified  viewers  using  aircraft  maintainers  from  the  653  Combat  Logistics  Support  Squadron  at 
Robins  AFB,  GA.  The  first  step  consisted  of  building  a  3D  computer  model  of  Avionics  Bay  3R 
of  the  USAF  F-15  aircraft.  Details  of  this  process  are  given  in  the  section  titled  "3D  MODEL 
CREATION  PROCESS."  The  maintainers  at  Robins  AFB  were  then  asked  to  perform  a 
simulated  maintenance  task  involving  Avionics  Bay  3R.  This  phase  of  the  project  was  designed 
to  integrate  the  information  obtained  from  the  literature  review,  CTA,  and  format  evaluations  by 
providing  the  maintainers  the  opportunity  to  interact  with  the  3D  F-15  model  of  the  avionics  bay. 
By  using  the  identified  viewers,  maintainers  could  evaluate  the  use  of  3D  models  for 
maintenance  tasks  in  general  and  the  effectiveness  of  these  two  “best  value”  viewers  in 
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particular.  Extensive  data  from  the  maintainers  was  gathered  during  the  study.  Details  and  results 
of  the  study  are  given  in  the  section  titled  "COMPARATIVE  STUDY." 

This  report  concludes  with  a  summary  and  discussion  of  directions  for  further  research. 
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BACKGROUND  FOR  EVALUATION  OF  3D  GRAPHICS 


Literature  Review  of  3D  Graphics  Human  Factor  Issues 

Interpretation  and  Perception  of  3D  Graphics  on  2D  Displays 

A  review  of  current  literature  was  performed  to  determine  the  usefulness  of  3D  graphic 
imagery  given  the  limitation  of  using  a  standard  2D  display.  This  literature  is  reviewed  below. 
Some  aspects  of  the  review  may  not  be  directly  applicable  to  the  CTA,  file  format  evaluation, 
and  comparative  study  phases  of  the  current  project,  but  are  mentioned  for  completeness  and 
future  reference. 

While  “true”  binocular  3D  stereo  displays  are  available  (i.e.,  displays  that  use  disparity 
information  to  present  objects  that  appear  to  float  in  space),  they  are  expensive  and  usually 
require  special  glasses  or  expensive  hardware.  Therefore,  standard  LCD  or  CRT  monitor  displays 
are  preferred  for  general  applications  such  as  most  maintenance  tasks. 

It  has  been  shown  that  the  use  of  graphics  and  visual  models  can  significantly  increase  the 
speed  and  accuracy  of  information  transfer,  especially  when  the  properties  and  limitations  of 
visual  perception  are  taken  into  account  (Haber  &  Wilkinson,  1982).  Three-dimensional  surface 
models  can  increase  a  user’s  ability  to  discern  information  by  adding  global  information  that  may 
not  be  obvious  from  2D  orthographic  projections  and  also  by  shifting  some  of  the  user’s 
cognitive  load  to  the  perceptual  system  (Robertson,  Mackinlay  &  Card,  1991).  Furthermore,  3D 
models  can  add  this  global  information  without  masking  individual  part  identity  (Pomerantz, 
1986). 

One  concern  about  the  use  of  3D  graphics  is  that  users  may  have  different  levels  of  ability 
for  spatial  reasoning  that  may  affect  their  ability  to  use  and  understand  3D  graphics.  While  this 
may  be  true  for  some  new  users,  it  has  been  shown  that  through  learning  and  practice,  this  ability 
can  be  improved  (McCuistion,  1991;  Osbom  &  Agogino,  1992;  Wiley,  1990;  Wiebe,  1991). 
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Monocular  Depth  Cues.  A  primary  attribute  for  perceiving  depth  is  stereopsis,  wherein 
the  two  eyes  receive  slightly  different  views  of  an  object  and  the  disparity  of  edges  and  textures 
gives  a  clue  to  their  location  in  depth.  However,  even  without  binocular  stereo  information,  there 
are  many  other  monocular  cues  to  depth,  including  size,  linear  perspective,  occlusion,  motion 
parallax,  texture  gradients,  shadowing,  contrast,  clarity,  and  brightness  (see  Wickens,  Todd  & 
Seidler,  1989;  Majchrzak  et  al.,  1987,  pg.  132-136).  Texture,  contrast,  clarity,  and  brightness 
tend  to  add  little  to  depth  perception  while  requiring  large  computational  costs.  Furthermore, 
because  of  possible  ambiguities  in  perception,  size  and  shadowing  effects  can  actually  cause 
misinterpretations  of  depth.  Therefore,  the  most  salient  monocular  depth  cues  are  motion, 
perspective,  and  occlusion.  These  alone  have  been  shown  to  overcome  a  lack  of  stereopsis  to 
give  powerful  cues  to  three-dimensionality  (Wickens,  Todd  &  Seidler,  1989;  Zhai,  Buxton  & 
Milgram,  1996). 

Motion  is  especially  useful,  as  it  can  be  used  to  unambiguate  2D  projections  of  3D  forms 
by  using  object  displacement  to  give  information  about  surface  segmentation  (Kaiser  &  Proffitt, 
1989).  It  also  allows  the  perceptual  system  to  track  parts  of  objects  and  relationships  between 
these  parts  because  of  object  constancy,  and  the  user  can  keep  these  relationships  assimilated 
even  after  the  object  stops  moving  (Robertson,  Mackinlay  &  Card,  1991).  Furthermore,  while 
motion  can  aid  in  the  initial  identification  of  objects  and  object  parts,  further  manipulation  of  that 
object’s  orientation  does  not  deleteriously  affect  the  identification  of  that  object  (Biederman  & 
Gerhardstein,  1993).  In  addition,  see  Jolicoeur,  1992,  who  cites  studies  that  show  that  rotation  of 
objects  with  a  standard  orientation,  like  faces  or  text,  etc.,  does  reduce  performance.  The  ability 
to  actively  manipulate  3D  object  models  on  2D  displays  provides  for  faster,  more  accurate  object 
identification  than  when  limited  to  static  graphics  (however,  it  should  be  noted  that  for  abstract 
data  visualization  —  as  opposed  to  concrete  object  perception  motion  gives  no  significant 
performance  benefits  over  static  3D  perspective  representations  (Wickens,  Merwin  &  Lin, 
1994)). 

Geometric  Modeling  —  Wireframes.  Solid  Models,  and  Shading.  There  is  little 
consensus  on  the  precise  terminology  used  for  geometric  models.  However,  for  purposes  of 
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clarity,  we  will  consolidate  the  images  used  in  the  reviewed  studies  into  four  basic  model  types: 
wireframes,  line  drawings,  surface  models,  and  solid  models. 

Straight  wireframe  models  simply  use  thin  interconnected  lines  to  represent  the  edges  of 
the  object  and  appear  as  see-through  outlines  of  the  objects.  Wireframes  have  disadvantages 
because  they  do  not  show  the  surfaces  of  the  objects  and  ambiguous  representations  can  occur 
(Groover,  1992).  Use  of  dashed  lines  or  semi-transparency  (reduced  contrast)  for  lines  that  would 
be  obscured  in  real  objects  helps  reduce  but  not  eliminate  ambiguity  (Zhai,  Buxton  &  Milgram, 
1996).  Line  drawings  are  wireframes  that  use  hidden  line  removal  (HLR),  or  the  removal  of  lines 
that  would  be  obscured  by  other  portions  of  the  real  object.  Line  drawings  also  help  to  alleviate 
some  of  the  ambiguity  of  wireframes  but  still  lack  shading  and  texture  elements  and  poorly 
represent  smooth  curvature  of  surfaces.  Surface  models  have  shading,  texture,  and  shadowing, 
but  are  still  represented  as  wireframes  or  polygonal  surfaces  in  the  computer.  Finally,  solid 
models  are  constructed  and  represented  as  solid  3D  shapes  in  the  computer  and  have  mass  and 
density  properties  consistent  with  the  object  materials  (Groover,  1992).  Many  visual  studies  use 
the  term  “solid  model”  when  they  are  really  referring  to  a  surface  model.  There  are  definite 
benefits  of  solid  models  for  CAD/CAE/CAM  applications  (e.g.,  the  ability  to  calculate  weight 
and  center-of-gravity  of  objects  and  groups  of  objects),  however,  these  benefits  are  not  necessary 
when  3D  object  models  are  simply  being  used  for  visualization. 

The  advantages  and  disadvantages  of  wireframe,  line  drawings,  and  surface  models  is  in 
terms  of  their  ease  of  relating  depth  and  their  perceived  realism.  While  complex,  full  color, 
shaded  renderings  provide  the  most  true-to-life  representation  of  objects,  simple  line  drawings  of 
the  same  objects,  although  appearing  less  realistic,  can  be  identified  as  quickly  and  accurately 
(Biederman  &  Ju,  1988).  The  question  becomes  how  much  and  what  type  of  information  should 
be  included  in  renderings  of  3D  objects  to  yield  good  performance  and  realism  without  resulting 
in  overkill. 

Subjective  ratings  of  perceived  realism  and  the  amount  of  geometric  information 
conveyed  have  shown  that  wireframe,  line  drawings,  and  shaded  surface  models  all  adequately 
convey  information  about  geometric  content,  but  shaded  surface  models  are  judged  to  be  the 
most  realistic  (McWhorter,  Hodges  &  Rodriguez,  1991).  Mental  rotation  tasks  (in  which  subjects 
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are  asked  whether  two  views  of  a  3D  object  are  of  the  same  or  different  objects)  also  show  that 
shaded  surface  models  give  significantly  better  results  (in  terms  of  reaction  times  and  error  rates) 
than  wireframe  or  line  drawings  (Barfield,  Sandford  &  Foley,  1988).  This  study  also  found  that 
the  addition  of  a  second  source  to  the  lighting  model  significantly  improved  perceived  realism 
while  manipulations  in  the  type  of  shading  had  no  effect.  This  indicates  that  computing  power 
may  be  better  spent  on  lighting  cues  than  on  shading. 

Human  Factors  Aspects  of  the  Graphic  User  Interface 

Many  aspects  of  the  GUI  are  relevant  to  the  display  and  manipulation  of  3D  images  and 
merit  brief  mention.  These  include  the  effective  field-of-view  of  the  graphic  window,  user 
manipulation  of  3D  objects  using  conventional  controls,  and  screen  attributes  such  as  screen 
design  and  clutter.  A  final  aspect,  the  use  of  color,  will  be  dealt  with  to  a  greater  degree  due  to  its 
complex  perceptual  nature. 

Geometric  Field-of-View.  The  geometric  field-of-view  (GFOV),  sometimes  called  the 
scene  field-of-view,  is  defined  as  the  field-of-view  of  the  computer’s  eye  for  the  viewport  of  the 
graphics  (Barfield,  Lim  &  Rosenberg,  1990).  This  may  or  may  not  (and  usually  does  not) 
coincide  with  the  user’s  field-of-view  in  terms  of  the  angular  aspect  of  the  display  monitor  at  the 
user’s  viewing  distance.  For  most  applications  that  display  simple  objects,  the  question  of  GFOV 
is  not  relevant;  however,  certain  3D  images  that  relate  to  large-scale  relationships  of  objects  on 
real-world  assemblies,  such  as  images  of  large  portions  of  a  wing  section  or  other  aircraft 
assemblies,  may  benefit  from  optimal  GFOV  parameters.  No  study  has  looked  at  these 
parameters  for  3D  object  displays  per  se,  but  in  studies  of  other  applied  spatial  tasks,  it  has  been 
found  that  the  optimal  GFOV  for  accuracy  of  directional  judgments  is  that  which  matches  the 
display  FOV  at  the  user’s  eye  (Barfield,  Lim  &  Rosenberg,  1990;  McGreevy  &  Ellis,  1986)  and 
has  an  eye-point  elevation  of  between  15°  and  45°  (Hendrix  &  Barfield,  1997). 

Interactive  Cnntrnl  of  3D  Objects.  The  presentation  of  3D  objects  on  2D  displays  also 
leads  to  questions  of  how  users  will  interact  with  those  objects.  To  gain  the  full  benefit  of  3D 
object  modeling,  objects  need  to  be  manipulated  in  different  ways  such  as  translation,  rotation 
about  any  axis,  and  size  scaling.  Translation  would  be  useful  to  shift  parts  vertically  or 
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horizontally  to  avoid  occlusion  and  rotation  would  provide  views  of  previously-hidden  object 
surfaces.  The  need  for  a  zooming  (or  size  scaling)  control  is  due  to  the  limited  resolution  and 
screen  size  of  the  display.  Certain  aspects  of  a  perspective  drawing  may  need  to  be  magnified  or 
“exploded”  to  visualize  individual  parts,  wires,  or  connections. 

While  detailed  descriptions  are  beyond  the  scope  of  this  report,  it  should  be  noted  that 
conventional  controls  (such  as  a  mouse  or  trackball)  have  been  found  to  provide  adequate  control 
when  used  with  appropriately-designed  software  “virtual”  controls  such  as  slider  controllers  or 
more  specialized  designs  (Chen,  Mountford  &  Sellen,  1988;  Houde,  1992)  or  even  allow  for 
different  types  of  task-dependent  virtual  controllers  (Nielson  &  Olsen,  1986). 

Screen  Attributes.  Another  relevant  aspect  of  graphic  display  is  screen  design  (see  Liu, 
1997,  for  a  review).  Proper  use  of  indentation,  grouping,  and  size  help  reduce  clutter  and  make 
screens  easier  to  read.  Color,  highlighting,  and  blinking  can  reduce  search  time  for  finding 
relevant  items.  Borders  around  display  areas  have  been  found  to  aid  in  separating  elements  and 
reducing  the  effects  of  clutter  (Matin  &  Boff,  1988).  Furthermore,  proper  layout  of  the  GUI  into 
well-defined  sections  with  minimal  text  and  data  can  aid  in  user  manipulation  and  understanding 
of  spatial  images  (Osborn  &  Agogino,  1992). 

Environmental  Effects  on  Color  Active-Matrix  Liquid  Crystal  Displays.  A  user’s  ability 
to  discriminate  and  identify  a  display’s  graphic  image  is  a  function  of  the  environment  in  which 
the  display  is  viewed.  An  important  environmental  factor  is  ambient  illumination,  which  reduces 
a  display’s  luminance  and  chromatic  contrast  and  produces  a  visual  adapting  luminance  for  the 
user.  Lower  luminance  contrast  makes  differentiation  of  image  detail  more  difficult.  In  addition, 
lower  chromatic  contrast  results  in  chromatic  gamut  reduction  or  “washout”  (the  “purest”  colors 
are  less  saturated  due  to  mixing  with  ambient  light,  so  fewer  colors  can  be  produced). 

Most  notebook  and  wearable  computers  incorporate  active  matrix  liquid  crystal  displays 
(AMLCDs).  These  displays  are  well  suited  for  use  in  high  ambient  illumination  environments; 
since  they  reflect  less  than  1%  of  incident  ambient  illumination,  they  are  also  suitable  for  use  in 
maintenance  environments.  Krantz,  Silverstein,  and  Yeh  (1992)  investigated  the  effects  of 
ambient  illumination  and  adapting  luminance  on  an  AMLCD  for  a  spatial  discrimination  task. 
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The  study  found  ambient  illumination  up  to  -100,000  lux  (the  same  as  that  produced  on  a 
horizontal  plane  by  the  sun  at  50°  above  the  horizon  on  a  clear  day)  and  adapting  luminances  up 
to  -30,000  cdJm  (equivalent  to  a  light  colored  object  under  direct  sunlight)  had  no  effect  on  the 
task  when  display  luminance  was  at  least  180  cd/m^  (a  typical  laptop  LCD  can  produce  a 
luminance  of  -100  cd/m^).  However,  spatial  discriminations  for  displays  producing  less  than  180 
cdJrn  were  most  affected  by  the  adapting  luminance,  in  particular,  when  adapting  luminance 
exceeded  display  luminance  by  a  ratio  of  more  than  100:1.  Chromatic  gamut  reduction  for 
AMLCDs,  or  the  inability  to  produce  pure  colors  due  to  desaturation  from  ambient  illumination, 
varies  from  nearly  zero  at  -10,000  lux  ambient  illumination  (similar  to  that  produced  on  a 
horizontal  plane  by  white  clouds  at  mid-day)  to  about  40%  at  -100,000  lux.  Therefore,  to  assure 
maximum  legibility,  users  should  orient  their  display  to  minimize  incident  light,  ambient 
illumination,  and  adapting  luminance. 

Finally,  it  should  be  noted  that  some  characteristics  of  AMLCD  hardware  may  be 
responsible  for  use  limitations  during  certain  maintenance  tasks.  While  AMLCDs  compare 
favorably  with  most  other  display  technologies  (plasma,  vacuum  fluorescent,  and 
electroluminescent)  in  terms  of  display  attributes  such  as  pixel  density,  screen  resolution,  and 
contrast,  AMLCDs  can  satisfactorily  operate  only  across  a  relatively  narrow  temperature  range 
(MacDonald  &  Lowe,  1997).  The  U.S.  Congress  Office  of  Technology  Assessment  (1995) 
reports  that,  “low  temperatures  affect  LCDs  primarily  in  two  ways:  1)  a  reduction  in  response 
time  and  color  gamut  shift  necessitates  the  need  for  heaters  and  programmable  look-up  tables, 
and  2)  the  backlight  required  for  an  LCD-based  display  must  incorporate  heaters  in  order  to  turn 
on  at  cold  temperatures.  In  addition,  liquid  crystal  materials  and  polarizers  often  suffer 
irreversible  damage  at  temperatures  above  90°C.” 

Color  in  Information  Display.  Color  is  an  important  dimension  of  graphics  visualization. 
Advantages  pf  using  color  include  the  ability  to  convey  realism,  designate  or  emphasize, 
segment,  warn/signify,  group,  imply  states,  and  improve  aesthetics  (Galitz,  1997;  Heath  & 
Flavell,  1985;  Travis,  1991).  For  example,  in  a.  medical  application,  three-dimensional  images 
were  more  easily  interpreted  when  the  various  transparent  overlapping  elements  were  colorized 
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(Brown,  Earnshaw,  Jern  &  Vince,  1995).  Also,  Wiebe  (1991)  found  that  long-term  study  of 
complex  spatial  relationships  and  visualization  of  global  form  were  enhanced  by  using  color. 
However,  if  used  improperly,  color  introduces  problems  such  as  distractions,  perceptual 
phenomena  which  vary  according  to  viewing  and  stimulus  conditions,  errors  or  ambiguities  for 
color  anomalous  (or  “color  blind”)  observers,  information  overload,  and  conflicting  cultural 
and/or  historical  messages  (Brown  et  al,  1995;  Horton,  1991;  Galitz,  1997). 

Many  principles  and  guidelines  exist  for  designing  color  into  information  displays.  Travis 
(1991)  suggests  two  general  design  principles.  First,  design  the  interface  for  a  monochrome 
display.  Color  should  be  a  redundant  coding  dimension  and  used  only  in  cases  where  there  is  a 
clear  requirement.  Second,  colors  should  be  consistent  with  the  user’s  mental  model.  Color 
coding  “systems”  will  be  effective  only  when  the  colors  match  what  the  user  expects  to  see  (the 
colors  are  appropriate  for  the  given  context). 

Several  reports  address  color  design  guidelines  such  as  recommended  applications, 
combinations  to  avoid,  and  size  and  orientation  constraints  (Murch,  1987;  Post,  1997,  Travis, 
1991).  Large  hue  differences  should  be  used  for  qualitative  information  (to  delineate  categories) 
while  luminance  and  saturation  differences  (or  small  hue  differences)  are  preferable  for 
representing  quantitative  data  (options  or  state  levels)  (Heath  &  Flavell,  1985).  Color  should  not 
be  used  for  an  arbitrary  ordinal  scale,  because  color  does  not  scale  uniformly  (Kaiser  &  Proffitt, 
1989).  However,  for  color  object  coding,  perceptual  and  cognitive  studies  have  shown  that  when 
the  user  must  perform  an  absolute  identification  task,  the  number  of  colors  differing  only  in  hue 
should  not  exceed  eleven  (and  should  be  the  appropriate  eleven),  or  confusion  will  result  (Post, 
1997;  Walraven,  1992).  Color  discrimination  varies  between  and  within  users,  therefore  color 
differences  must  be  large  enough  to  ensure  that  most  users  can  differentiate  them.  Color 
discrimination  and  luminance  contrast  decreases  with  increasing  peripheral  eccentricity, 
requiring  larger  and  higher  contrast,  respectively,  for  peripherally  presented  visual  information. 

Three-dimensional  effects  in  computer-aided  design  (CAD)  and  other  3D  graphic 
applications  can  be  improved  by  using  blue  for  distinct  objects  and  red  for  close  objects  (Brown 
et  al,  1995).  Intermediate  distances  are  represented  by  intermediate  hues  such  as  green,  yellow, 
and  orange.  A  table  of  the  best  and  worst  foreground/background  color  combinations  for  text  and 
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panels,  assessed  via  subjective  preference  and  consistent  with  color  phenomena  discussed  below, 
can  be  found  in  Brown  et  'al.  (1995).  Finally,  highly  saturated  colors  can  cause  visual  fatigue  as 
well  as  induce  many  of  the  color  artifacts  described  below. 

An  additional  color  design  consideration  is  that  color  coding  may  be  inappropriate  for 
nearly  10%  of  the  population  (mostly  male)  who  are  color  anomalous  (commonly  called  “color 
blind”).  Most  color  deficient  observers  confuse  red,  yellow,  orange,  and  green,  although  a  very 
small  percentage  confuse  blue,  cyan,  yellow  and  white  (Post,  1997).  Adapting  color  display 
information  for  color  anomalous  observers  requires  additional  design  guidelines  (Walraven, 
1992).  First,  color  confusion  is  most  problematic  when  there  is  no  perceived  chromaticity  or 
luminance  difference.  Higher  luminance  contrast,  which  takes  into  account  a  defective  observer  s 
lower  luminance  capability  for  colors  associated  with  the  defective  system,  can  help  differentiate 
objects.  Second,  color  objects  subtending  small  visual  angles  are  more  difficult  to  discriminate 
and  should  be  avoided,  especially  when  desaturated  and/or  of  low  luminance.  Large  areas, 
however,  may  be  discriminated  relatively  well.  Third,  red/green  defectives  rely  more  on  blue  and 
yellow  discriminations,  so  these  colors  should  be  included  in  the  design.  Finally,  the  existence  of 
color  defectives  emphasizes  the  fact  that  color  should  be  a  redundant  coding  dimension  (used  in 
combination  with  luminance,  shape,  size,  etc.)  and  not  the  only  means  of  identification. 

Inappropriate  color  usage  in  information  displays  can  lead  to  numerous  undesired  visual 
phenomena  and  artifacts  (Horton,  1991;  Murch,  1987;  Post,  1997;  Walraven,  1992).  Adjacent 
colors  can  produce  simultaneous  color  contrast  (shift  in  perceived  hue)  and  therefore  should  be 
separated  spatially.  The  human  lens  does  not  focus  all  wavelengths  exactly  on  the  retina.  Blues 
focus  slightly  in  front  of  the  retina  and  reds  focus  slightly  behind  the  retina  (chromatic 
aberration),  so  that  blue  and  red  objects  sometimes  appear  blurry.  Simultaneous  presentation  of 
reds  and  blues  may  cause  them  to  appear  in  different  depth  planes  (chromostereopsis).  Chromatic 
aberration  and  chromostereopsis  increase  with  increasing  saturation.  Equally  bright  colors  may 
not  be  distinguishable,  so  redundant  luminance  coding  is  often  necessary.  Intense  or  large  color 
areas  may  produce  color  aftereffects  (McCullough  effect).  Finally,  the  hue  appears  to  shift  with 
increasing  or  decreasing  luminance  (Bezold-Brucke  effect). 
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In  summary,  color  may  enhance  3D  model  visualization  for  maintenance  tasks,  although 
it  should  be  used  cautiously  and  in  accordance  with  the  aforementioned  guidelines, 
recommendations,  and  caveats.  For  the  maintenance  task  under  study,  color  will  be  most 
beneficial  to  help  differentiate  objects.  The  use  of  color  as  an  absolute  identifier  (trying  to 
reproduce  the  exact  color  of  real  color-coded  wires)  is  not  recommended,  since  color  fidelity  is  a 
function  of  the  environment,  display  system,  and  the  user’s  visual  system.  The  use  of  highly 
saturated  colors  should  be  minimized,  but  this  may  ultimately  be  determined  by  the  degree  of 
realism  required.  Finally,  the  design  should  take  into  account  possible  color  desaturation  and 
contrast  reduction  effects  due  to  ambient  illumination,  the  existence  of  color  anomalous  users, 
and  potential  visual  artifacts. 

3D  Graphics  for  Maintenance  Tasks 

Few  studies  have  been  conducted  on  the  applicability  of  3D  graphics  directly  to 
maintenance  tasks.  There  is  at  least  one  instance  in  the  general  literature  of  placing  computerized 
manuals,  wiring  diagrams  and  documentation  on  laptop  computers  for  use  by  maintenance 
workers  (Rankin,  Allen,  Sargent  &  Graeber,  1997);  however,  little  is  known  about  the  efficacy  of 
this  approach.  An  attempt  was  made  in  this  literature  review  to  analyze  the  use  of  3D  graphics  as 
a  general  means  of  relaying  information  to  the  viewer.  In  this,  it  was  shown  that  3D  graphics  are 
a  powerful  tool  for  quickly  and  accurately  identifying  parts,  objects,  and  assemblies  of  objects 
and,  therefore,  should  prove  useful  when  integrated  into  systems  to  be  used  by  maintenance 
technicians. 


Cognitive  Task  Analysis  Background 
Review  of  Current  Literature  and  Techniques 

Cognitive  Task  Analysis  is  most  often  associated  with  several  knowledge  elicitation 
techniques  such  as  knowledge  acquisition  with  question  probes  and  conceptual  graph  structures 
(“concept  mapping”),  observations,  interviews,  task  analysis,  protocol  analysis,  and  structural 
analysis.  CTA  is  actually  an  extension  of  task  analysis,  where  the  objective  is  to  determine  the 
cognitive  abilities  necessary  to  perform  a  task.  Traditional  task  analysis  accesses  only  the 
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physical,  generally  observable,  aspects  of  a  task,  whereas  CTA  quantifies  or  represents  the 
underlying  cognitive  processes  that  are  not  directly  observable,  thus  providing  an  indication  of 
cognitive  workload,  information  processing,  and  decision  making  associated  with  a  task. 

The  knowledge  elicitation  techniques  that  comprise  a  CTA  are  not  applied  in  any  set 
order,  but  rather  vary  with  the  application.  CTA’s  flexibility  is  reflected  in  its  definition,  which 
often  varies  among  experts.  One  common  definition  states  that  CTA  is  a  knowledge  acquisition 
procedure  that  involves  identifying  the  knowledge  and  the  cognitive  skills  required  to  perform  a 
task  at  acceptably  high  levels  (Gordon,  Schmierer,  &  Gill,  1993).  Thus,  according  to  this 
definition,  CTA  is  used  to  identify  and  solve  cognitive  issues  that  occur  while  performing  a  task. 
Relatively  few  boundaries  exist  when  using  CTA  for  tasks  involving  cognitive  elements. 

Three  approaches  have  been  recommended  for  the  majority  of  CTA  applications  (Gordon 
&  Gill,  1994).  The  first  approach  relies  on  a  formal  analysis  of  the  application  to  uncover  the 
cognitive  demands  of  the  task  for  a  specific  environment  (assessing  the  tasks  involved).  A  second 
approach  relies  on  empirical  techniques  to  analyze  how  humans  perform  a  task  (assessing  the 
procedure).  The  empirical  approach  is  particularly  suitable  for  developing  training  requirements 
for  existing  systems.  The  third  approach  relies  on  a  computer  model  that  simulates  the  cognitive 
activities  involved  in  completing  a  task  (modeling  the  tasks  and  procedure).  Applications  of  CTA 
often  involve  a  mixture  of  all  three  methods  (Roth  and  Mumaw,  1995). 

Benefits  of  CTA 

Concept  mapping  and  semi-structured  interviews  were  the  two  main  techniques  used  to 
formulate  the  CTA.  Concept  mapping  involves  using  a  graphical  structure  to  display  and 
organize  the  gleaned  knowledge  and  relationships.  Semi-structured  interviews  use  question  and 
answer  dialogs  to  obtain  and  qualify  information  (the  section  titled  "COGNITIVE  TASK 
ANALYSIS"  contains  more  information  on  defining  and  applying  concept  maps  and  semi- 
structured  iriterviews).  These  two  specific  knowledge  elicitation  techniques  have  several 
advantages,  particularly  when  applied  concurrently  within  a  session. 

There  are  several  advantages  in  using  graphical  structures  via  concept  maps  to  represent 
the  process  of  knowledge  acquisition  (Gordon  &  Gill,  1992).  The  graphical  structure  is  depicted 
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on  a  whiteboard  in  front  of  all  interview  participants.  Thus,  potential  ambiguities  in  interpreting 
knowledge  concepts  and  their  relationships  are  minimized.  This  allows  the  subject  matter  expert 
(SME)  and  the  analyst  to  easily  identify  missing,  inconsistent,  or  redundant  information. 
Flexibility  in  the  content  and  type  of  probing  questions  allows  the  analyst  to  structure  the 
interview  in  real-time,  as  needed  -  an  essential  element  of  a  concept  map.  Most  importantly,  the 
probes  allow  the  analyst,  rather  than  the  SME,  to  structure  the  interview.  This  is  critical  when 
attempting  to  gather  a  large  body  of  knowledge,  since  the  SME  can  easily  traverse  topics 
tangentially,  leading  to  excessive  data  that  may  be  irrelevant  to  the  topic  of  interest.  Additionally, 
SMEs  report  that  concept  map  interviews  are  interesting,  which  can  lead  to  more  complete 
knowledge  elicitation. 

Verbal  interviewing  is  one  of  the  most  common  knowledge  elicitation  techniques 
(Brenner,  Brown  &  Canter,  1985).  These  interviews  typically  focus  on  the  SME’s  recall  of  an 
event  or  past  experience  via  responses  to  verbal  questions.  These  sessions  are  usually  in  written 
(notes),  audio,  and/or  video  formats,  since  the  amount  of  information  precludes  one  person  from 
capturing  it  all.  The  audio  and  video  tapes  allow  the  analyst  the  opportunity  to  later  review  and 
verify  key  points  that  may  have  been  missed  in  the  notes. 

Meister  (1985)  cites  three  common  types  of  verbal  interviews:  1)  unstructured,  where  the 
SME  expounds  on  a  topic  relatively  unguided  by  the  analyst,  2)  semi-structured,  where  the  SME 
expounds  on  a  series  of  carefully-selected  questions,  and  3)  structured,  where  the  SME  responds, 
usually  by  a  forced  choice,  to  very  precise  questions.  Semi-structured  interviews  were  employed 
in  this  study  because  the  relative  freedom  of  the  SME’s  answers  would  help  to  uncover  topics 
relevant  to  the  project,  but  previously  unknown  to  the  analysts. 
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COGNITIVE  TASK  ANALYSIS 


Method 

Five  F-15  and  one  C-141  aircraft  maintainers  from  the  653  Combat  Logistics  Support 
Squadron  (CLSS)  served  as  the  SMEs.  The  F-15  aircraft  served  as  the  focus  because  of  its 
complicated,  compact  system  that  represents  most  of  the  difficulties  encountered  in  aircraft  when 
working  with  TO  graphics.  The  SMEs  were  chosen  to  help  identify  representative  tasks  in  which 
TO  graphics  play  vital  roles  and  to  evaluate  potential  3D  models. 

For  this  study,  the  empirical  CTA  approach  was  used.  The  CTA  was  implemented  using 
two  knowledge  elicitation  techniques:  concept  mapping  and  semi-structured  interviews.  Concept 
mapping  was  chosen  because  of  its  unique  ability  to  capture  knowledge;  that  is,  it  involves  a 
graphical,  interactive  interview  that  involves  SMEs  in  the  elicitation  process.  The  semi- 
structured  interview  provided  an  additional  and  more  in-depth  method  for  capturing  SME 
knowledge  through  a  series  of  probing  questions. 

The  653  CLSS  maintains  a  depot  of  Aircraft  Battle  Damage  Repair  (ABDR)  C-130,  C- 
141,  and  F-15  aircraft.  Their  primary  mission  as  a  highly  trained,  dedicated,  flexible,  and 
mission-ready  logistics  team  is  to  support  Air  Force  ABDR  operations.  Maintainers  are  trained  in 
three  areas:  ABDR,  major  maintenance  augmentation  (routine  repair),  and  deployable  Jet  Engine 
Intermediate  Maintenance  (JEM).  Six  maintainers  skilled  in  ABDR  and  routine  maintenance 
participated  in  the  CTA  study. 

Knowledge  Elicitation  Techniques 

The  methodology  used  to  collect  data  was  developed  specifically  for  this  study.  The 
techniques  included  informal  briefings,  questionnaires,  and  concept  mapping  and  semi-structured 
interviews. 

Informal  Briefing.  The  analyst  informally  briefed  each  maintainer  prior  to  the  CTA 
sessions.  The  briefing  contained  an  overview  of  the  CTA  study  and  objectives.  The  maintainer  s 
role  as  an  SME  dealing  with  TOs  was  also  established  at  this  time. 
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Questionnaire.  After  the  briefing,  each  maintainer  completed  a  background  information 
questionnaire  (Appendix  A).  There  were  separate  questionnaires  for  routine  and  ABDR 
maintenance  sessions.  The  purpose  of  the  questionnaires  was  to  gather  information  about  the 
maintainer’ s  aircraft  maintenance  experience  by  aircraft  type.  The  maintainer’ s  Air  Force 
Specialty  Code  (AFSC),  an  indication  of  experience  and  rank  were  also  recorded.  The 
maintainer’ s  background  information  provided  a  basis  from  which  the  research  team  could 
formulate  the  task  and  structure  probes  in  the  CTA. 

Concept  Mapping.  Concept  mapping  consisted  of  two  parts:  a  conceptual  phase  and  a 
solution  or  decision  phase.  In  the  conceptualization  of  a  task  or  problem,  the  SME’s  perceptions 
of  the  variables  involved  were  collected.  In  the  solution  or  decision  phase,  the  SME’s  responses 
to  the  variables  were  recorded  (McFarren,  1997).  In  studying  both  phases  together,  an  SME’s 
representation  of  the  problem  and  the  solution  were  recorded. 

The  concept  mapping  sessions,  using  a  verbal  protocol  analysis,  provided  specific 
information  on  the  use  of  TO  graphics  and  the  potential  use  of  3D  graphics  in  aircraft 
maintenance.  Verbal  protocol  analysis  can  be  reported  in  one  of  two  ways,  either  in  concurrent 
verbal  or  retrospective  reports.  These  reports  claim  to  reflect  cognitive  processes.  Concurrent 
verbal  reports  consist  of  a  “talk  aloud”  and  “think  aloud”  phase,  where  the  cognitive  processes 
are  verbalized  in  successive  states.  This  study  used  the  retrospective  verbal  report,  where  a 
memory  trace  of  the  information  required  to  complete  a  task  is  recorded.  Retrospective 
information  comes  from  memory  of  a  completed  project  and  may  expose  errors  or  omissions 
identifiable  only  in  “hindsight”  (Ericsson  &  Simon,  1993).  The  maintainers  were  asked  to 
vocalize  their  actions  as  if  performing  a  specified  task. 

Key  points  were  represented  as  nodes,  or  concepts,  within  a  particular  domain.  Each  node 
was  depicted  by  text  within  an  oval  and  represented  a  “piece”  of  information  collected  in  the 
elicitation  process.  Concepts  were  connected  through  relational  links  denoting  the  association 
between  tworconcepts  and  depicted  the  sequential  aspect  of  the  task.  The  meaning  contained  in 
any  particular  concept  map  was  primarily  a  function  of  the  relations  between  concepts.  As  seen 
in  Figure  1,  the  top-level  nodes  “Scan  of  Cockpit  Without  Removing  Items,”  “Search  Areas 
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Difficult  to  Reach,”  and  “Search  Additional  Areas”  represents  the  steps  involved  in  the  location 
of  Foreign  Object  Damage  (FOD)  in  a  cockpit. 


Semi-structured  Interviews.  As  already  mentioned,  the  content  of  the  semi-structured 
interviews  was  predetermined  and  the  sequence  and  length  of  responses  was  allowed  to  vary. 
The  semi-structured  interview  format  allowed  a  more  complete  coverage  of  the  SME  domain 
knowledge  than  may  be  covered  through  an  unstructured  interview  (Cooke,  1994).  All  of  the 
questions  addressed  points  regarding  the  requirements  of  3D  graphics  for  use  in  routine  and 
ABDR  maintenance.  These  questions  were  developed  from  an  in-depth  literature  review  of 
maintenance  activities  and  interviews  with  maintenance  personal  prior  to  the  CTA. 

Experimental  Procedure 

Each  maintainer  participated  in  two  CTA  sessions.  One  session  focused  on  routine 
aircraft  maintenance  while  the  other  session  focused  on  ABDR  maintenance.  During  the  first 
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session,  the  maintainer  was  briefed  on  the  study  (see  Informal  Briefing),  then  given  a 
background  information  questionnaire.  For  each  session,  concept  mapping  interviews  were 
conducted  followed  by  the  semi-structured  interview.  The  CTA  sessions  each  lasted  60  to  90 
minutes. 

A  team  of  three  researchers  participated  in  the  CTA  sessions.  During  the  concept 
mapping  interviews,  one  team  member  served  as  the  analyst  and  transcribed  the  map  onto  a  large 
whiteboard.  The  two  other  team  members,  along  with  the  analyst,  guided  the  maintainer  through 
a  representative  task  by  asking  question  probes. 

In  both  CTA  sessions,  routine  and  ABDR,  the  maintainers  were  asked  to  “walk  through” 
a  pre-selected  aircraft  maintenance  task.  The  scenarios  for  routine  maintenance  included: 

•  Foreign  Object  Damage 

•  Brake  Change 

•  Inspection  by  Feel 

•  Perspective  Problem 

The  scenarios  for  ABDR  maintenance  were  specific  to  each  maintainer’ s  repair  specialty. 
Three  different  specialties  were  encountered: 

•  Crew  Chief 

•  Electrician 

•  Sheet  Metal 

Appendix  B  contains  a  detailed  explanation  of  all  seven  scenarios.  Once  a  scenario  had 
been  explained,  the  maintainer  was  asked  to  break  the  task  down  into  three  to  five  steps.  These 
steps  formed  the  top-level  nodes  for  the  concept  map  and  the  maintainer  was  asked  to  develop 
each  concept.' There  were  no  restrictions  placed  on  the  order  in  which  the  maintainers  developed 
the  nodes.  As  the  maintainer  spoke,  the  analyst  drew  the  map  on  a  large  whiteboard  in  front  of 
the  maintainer  and  the  research  team.  As  part  of  the  inherent  validity  in  the  concept  mapping 
procedure,  the  maintainer  was  encouraged  to  make  changes  and  clarifications  to  the  map 
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whenever  appropriate.  The  session  ended  when  the  maintainer  felt  he  had  completely  represented 
his  actions  during  task. 

After  completing  the  concept  map,  the  CTA  transitioned  to  a  semi-structured  interview. 
The  maintainers  were  asked  a  series  of  questions  categorized  into  two  groups:  (1)  Do  the  current 
graphics  in  the  TOs  help  you  perform  your  maintenance?  and  (2)  Would  3D  graphics  aid 
maintenance?  The  goal  of  the  interview  session  was  to  gather  the  maintainer’ s  opinion  of  the 
current  TO  graphics  and  the  potential  use  of  3D  graphics.  The  questions  were  repeated  for  each 
of  the  maintainers.  Although  these  questions  were  established  before  the  session  began,  slight 
modifications  were  made  to  incorporate  information  gathered  from  the  concept  maps.  The  entire 
CTA  session  was  audio-  and  video-recorded  to  ensure  no  loss  of  information. 

Subjects 

Six  aircraft  maintainers  from  the  653  CLSS,  stationed  at  Robins  AFB,  Warner  Robins 
GA,  participated  in  the  study.  Skill  levels  ranged  from  3-level  to  7-level.  One  of  the  maintainers 
did  not  have  maintenance  experience  with  the  F-15  aircraft,  but  rather  had  C-141  maintenance 
experience.  All  participants  had  routine  and  ABDR  aircraft  maintenance  experience.  On  average, 
the  maintainers  had  eight  years  F-15  routine  maintenance  and  four  years  F-15  ABDR 
maintenance  experience. 


Data  Analysis 

The  raw  data  consisted  of  the  research  team’s  notes,  audio  and  video  recordings  of  all 
information  sheets,  concept  maps,  and  semi-structured  interview  sessions.  Over  150  pages  of 
transcribed  data  were  compiled.  The  data  set  was  used  to  clarify  and  complete  the  concept  maps 
and  to  categorize  16  areas  of  interest  (topics)  identified  in  the  semi-structured  interview. 

Concept  Map  Analysis 

The  concept  maps  provided  detailed  information  regarding  the  use  of  current  TO  graphics 
within  the  context  of  routine  and  ABDR  repairs.  Direct  observation  of  a  maintenance  task  could 
have  added  validity  to  the  concept  mapping  data  and  identified  any  additional  difficulties 
encountered  by  the  maintainer.  However,  observation  was  not  possible  due  to  time  constraints. 
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Only  a  retrospective  verbal  report  could  be  obtained.  Two  maps  were  created  for  each 
maintainer:  one  regarding  routine  maintenance  and  one  regarding  ABDR  maintenance.  The 
research  team  made  special  notes  whenever  graphics  were  referenced. 

The  information  in  the  maps  depicted  the  importance  of  and  the  order  in  which  the  TO 
graphics  were  referenced  for  a  particular  task.  The  maps  will  be  discussed  in  two  groups;  routine 
and  ADBR.  Four  unique  tasks  were  mapped  in  the  routine  sessions  (FOD,  brake  change, 
inspection  by  feel,  and  perspective  problem).  For  the  FOD  scenario,  graphics  typically  were  not 
required  to  perform  maintenance.  A  novice  may,  however,  need  to  reference  a  graphic  to  become 
oriented  in  the  cockpit  and  locate  any  removable  items.  For  the  brake  change  scenario,  the  TO 
graphics  would  need  to  be  referenced  to  complete  a  tire  removal.  The  concept  map  for  this 
scenario  is  seen  in  Figure  2. 


Figure  2.  Brake  Change  Concept  Map. 


The  inspection  by  feel  scenario  dealt  with  bird  strikes  to  an  aircraft.  In  this  scenario,  the 
graphic’s  perspectives  were  so  poor  that  the  maintainers  could  not  determine  what  systems  were 
damaged  so  they  had  to  perform  the  inspection  by  feel.  In  the  perspective  problem  scenario,  two 
specific  tasks  were  illustrated:  a  fire  loop  problem  and  C-141  window  post  replacement.  For  both 
of  these  tasks,  a  novice  would  need  to  reference  TO  graphics  in  order  to  locate  the  positions  of 
the  fire  loops  and  window  posts.  In  these  scenarios,  the  TO  graphics  were  referenced,  but  they 
were  very  unclear  and  did  not  aid  the  maintainers.  Several  details,  such  as  bolts  and  seals,  were 
not  represented  in  the  graphics  for  the  fire  loops  and  window  post  assembly.  The  C-141  window 
post  replacement  scenario  is  shown  in  Figure  3. 
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Figure  3.  Window  Post  Replacement  Map. 

In  the  ABDR  sessions  four  unique  tasks  were  mapped  (crew  chief,  electrician,  and  sheet 
metal).  For  the  crew  chief,  the  overwhelming  trend  was  to  use  graphics  to  help  identify  damaged 
components  and  damage  that  may  be  hidden  from  view.  Both  novice  and  experienced 
maintainers  use  graphics  at  some  point  in  their  ABDR  crew  chief  duties.  Figure  4  shows  a 
typical  chew  chief  procedure. 

In  the  electrical  scenario  the  regular  maintenance  wiring  diagram  TOs  were  found  to  be 
more  useful  than  ABDR  TOs.  For  the  sheet  metal  scenario,  TOs  were  used  to  identify  damage 
that  travels  through  the  plane.  The  graphics  depict  each  layer  of  components  to  aid  in 
maintenance.  Where  applicable,  the  maintainers  rely  heavily  on  drawings  created  by  the 
engineers. 


Figure  4.  Crew  Chief  Procedure  Concept  Map. 
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Appendix  C  contains  all  six  of  the  maintainers’  concept  maps  for  both  the  routine  and 
ABDR  sessions.  Some  maps  showed  the  benefits  of  current  TO  graphics  while  other  maps  showed 
problems. 

Interview  Analysis 

To  organize  the  interview  data,  two  spreadsheets  were  constructed  to  represent  the 
maintainer’s  responses  to  the  16  topics.  One  spreadsheet  contained  responses  for  routine 
maintenance  while  the  other  was  for  ABDR  maintenance.  The  same  list  of  topics  was  used  for 
both  types  of  maintenance.  The  16  topics  are  listed  below: 

1 .  Do  TO  graphics  aid  in  maintenance?  How? 

2.  Is  there  ever  a  time  when  TO  graphics  do  not  aid  in  maintenance?  How? 

3.  Are  there  any  TO  graphics  that  are  inadequate  or  misleading?  When? 

4.  Does  experience  ever  override  misleading  graphics?  When? 

5.  Are  there  any  TO  graphics  improperly  drawn  or  mislabeled?  When? 

6.  Is  there  ever  a  time  when  you  reference  supplemental  material  because  of  poor 
graphics?  What?  When? 

7.  Would  TO  graphics  aid  a  novice?  When? 

8.  Is  there  ever  a  time  when  mental  rotation  of  a  TO  graphic  is  required  to  complete  a 
job?  When? 

9.  Was  there  ever  a  time  when  you  had  to  change  your  visualization  of  a  TO  graphic  to 
complete  a  job?  When? 

10.  Was  there  ever  a  time  when  graphics  were  unavailable?  When? 

11.  Would  3D  graphics  aid  in  typical/ABDR  maintenance?  When? 

12.  Would  3D  graphics  aid  a  novice?  When? 

13.  Which  TO  graphics  need  improvement? 

14.  What  type  of  improvement  is  needed  for  the  TO  graphics? 

15.  Do  you  ever  visualize  peeling  away  levels  of  a  surface?  When? 
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16.  Is  full  manipulation  of  3D  graphics  required? 


The  results  of  responses  to  the  questions  will  be  reported  separately  with  respect  to 
routine  and  ABDR  maintenance  and  are  summarized  below: 

Question:  Do  TO  graphics  aid  in  maintenance?  How? 

Routine  Maintenance:  All  of  the  maintainers  stated  that  TO  graphics  aided  in  performing 
aircraft  maintenance.  Without  the  graphics  they  would  have  to  reference  another  aircraft  or  rely 
on  the  experience  of  other  maintainers  to  complete  repairs.  They  also  believed  that  TO 
corrections  or  block  upgrades  to  aircraft  in  a  graphic  would  more  likely  be  recognized  than  in 
text. 


ABDR  Maintenance:  All  of  the  maintainers  stated  that  the  ABDR  TO  graphics  were 
useful,  but  there  was  room  for  improvement.  A  few  maintainers  suggested  improving  the 
organization  of  the  ABDR  TO  to  allow  for  easier  access  to  graphics.  Similarly,  another 
maintainer  stated  that  the  graphics  did  not  aid  visual  inspection  and  assessment  tasks,  due  mainly 
to  the  time  required  to  locate  the  graphic.  He  stated  that  time  is  crucial  in  ABDR  tasks  and  the 
quicker  a  graphic  is  found,  the  better.  All  agreed  that,  once  found,  graphics  were  helpful  in 
aiding  repairs. 

Question:  Is  there  ever  a  time  when  TQ  graphics  do  not  aid  in  maintenance?  How? 

Routine  Maintenance:  Only  two  maintainers  responded  “yes”  to  this  question.  One 
maintainer  commented  that  the  lack  of  detail  in  the  graphics  made  them  very  difficult  to  use.  The 
graphics  did  not  give  him  spatial  location  information  for  essential  components.  The  other 
maintainer  mentioned  a  problem  with  finding  parts  on  a  TO  graphic.  In  this  case,  the 
organization  of  the  TO  appeared  to  be  the  leading  cause  of  this  problem.  For  instance, 
instructions  pertaining  to  a  graphic  usually  precede  the  graphic  by  several  pages.  The  maintainer 
must  continually  refer  back  and  forth  between  the  text  and  graphic  and  integrate  the  information. 
This  exercise  often  results  in  confusion,  frustration,  and  lost  time. 
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ABDR  Maintenance:  None  of  the  maintainers  responded  to  this  question.  It  would  be 
incorrect,  however,  to  assume  a  “no”  response  indicates  TO  graphics  always  aid  in  ABDR 
maintenance.  Typical  TOs  primarily  focus  on  information  related  to  a  particular  task.  ABDR 
TOs  have  information  centered  on  physical  and  functional  means. 

Question:  Are  there  any  TO  graphics  that  are  inadequate  or  misleading?  When? 

Routine  Maintenance:  Although  only  a  few  of  the  maintainers  called  out  specific 
examples  of  inadequate  and  misleading  graphics,  all  stated  that  they  had  run  across  inadequate  or 
misleading  graphics  at  some  time  in  their  careers.  Graphics  for  the  hydraulic  lines,  electrical 
systems,  engine  bay  and  fuel  cells  were  deemed  particularly  inadequate  or  misleading. 

ABDR  Maintenance:  Opinions  differed  for  this  question.  Four  maintainers  said  they  had 
never  encountered  inadequate  or  misleading  graphics,  while  two  others  stated  they  had.  Of  the 
maintainers  that  had,  graphics  of  the  engine  bay  and  wings  provided  inadequate  and  misleading 
information.  Specifically,  the  liimted  perspective  and  lack  of  reference  points  in  the  engine  bay 
created  the  problems.  Typically,  graphics  are  deemed  inadequate  or  misleading  based  on 
perspective.  The  format  of  a  graphic  is  just  as  important  as  content. 

Question:  Does  experience  ever  override  misleading  graphics?  When? 

Routine  Maintenance:  All  except  one  maintainer  often  relied  on  experience  to  overcome 
a  misleading  graphic.  Neither  experience  nor  graphics  alone  can  provide  enough  information  for 
the  maintainers  to  complete  every  possible  repair.  However,  when  choosing  between  a  TO 
graphic  that  does  not  look  accurate  and  a  mental  representation  based  on  experience,  the 
maintainers  chose  their  recollections.  Of  course,  this  begs  the  question  whether  a  maintainer 
could  deem  a  graphic  misleading  unless  there  was  prior  knowledge  to  draw  upon. 

ABDR  Maintenance:  One  maintainer  had  not  discovered  many  misleading  graphics,  but 
felt  that  a  potential  source  of  problems  existed  with  the  wing  graphics.  The  plane  s  orientation  in 
the  graphics  :is  often  unknown.  In  this  case,  experience  and  physical  cues  help  the  maintainers 
determine  the  graphic’s  perspective. 


25 


Question:  Are  there  any  TO  graphics  improperly  drawn  or  mislabeled?  When? 

Routine  Maintenance:  Two  of  the  six  maintainers  found  some  graphics  to  be  inaccurate 
(improperly  drawn).  The  major  source  of  this  inaccuracy  resulted  from  the  lack  of  realism 
between  the  components  as  seen  in  the  graphics  and  on  the  plane,  again  found  in  the  graphics  of 
the  wing. 

ABDR  Maintenance:  None  of  the  maintainers  found  any  of  the  ABDR  graphics  to  be 
improperly  drawn  or  misleading. 

Question:  Is  there  ever  a  time  when  you  reference  supplemental  material  because  of  poor 
graphics?  What?  When? 

Routine  Maintenance:  All  six  of  the  maintainers  claimed  to  have  referenced  supplemental 
material  when  graphics  were  poor.  The  most  popular  “supplement  was  a  similar  aircraft.  When 
a  graphic  was  difficult  to  use,  the  maintainers  preferred  to  look  at  another  aircraft  to  get  the 
necessary  information;  a  process  that  offers  more  perspectives  and  a  larger  field  of  view  than  can 
be  found  in  TO  graphics.  This  technique  was  usually  employed  while  working  on  the  wings  and 
deep  within  a  bay.  One  reason  the  graphics  may  be  less  effective  is  that  these  locations  are 
shown  at  unusual  perspectives  and  thus  are  difficult  to  comprehend. 

ABDR  Maintenance:  Only  two  maintainers  said  they  referenced  supplemental  material. 
Two  maintainers  did  not  respond.  In  this  case,  supplemental  material  was  another  aircraft. 
Routing  of  wiring  systems  and  gaining  a  proper  perspective  were  two  examples  requiring 
supplemental  information. 

Question:  Would  TQ  graphics  aid  a  novice?  When? 

Routine  Maintenance:  All  the  maintainers  felt  that  novices  would  need  to  reference  a 
graphic  to  aid  their  completion  of  the  repairs.  The  point  was  also  made  that  graphics  serve  as  a 
training  tool.  Although  the  graphics  may  be  misleading  and  incomplete  in  some  cases,  they 
provide  more  information  than  can  be  obtained  without  them.  Obviously,  some  repairs  may 
require  even  more  instruction  than  can  be  found  in  the  TO.  In  these  instances  novices  often  seek 
help  from  experts. 
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ABDR  Maintenance:  All  of  the  five  maintainers  that  responded  agreed  that  novices 
would  benefit  from  having  graphics.  Because  of  the  additional  skills  required  to  perform  ABDR 
maintenance,  the  use  of  graphics  by  novices  is  essential.  The  TO  graphics  would  be  most 
beneficial  when  performing  small  tasks.  On  larger  repairs,  the  complexity  involved  exceeds  any 
help  that  the  current  graphic’s  format  can  provide,  so  novices  must  seek  additional  assistance. 

Question:  Is  there  ever  a  time  when  mental  rotation  of  a  TO  graphic  is  required  to  complete 
a  job?  When? 

Routine  Maintenance:  Four  maintainers  commented  that  mentally  rotating  graphics  is 
required  to  complete  some  jobs.  For  instance,  when  working  on  wiring  harnesses,  the  graphics’ 
perspectives  were  oriented  differently  that  the  corresponding  aircraft  part.  As  a  result,  the 
maintainers  perform  a  mental  rotation  of  the  graphic  to  provide  consistency  with  the  part’s 
physical  orientation.  According  to  maintainers,  the  perspectives  of  the  graphics  are  not  always 
useful  and  may  be  misleading.  Confusion  often  results  when  the  perspective  of  the  graphic  does 
not  match  the  perspective  of  the  maintainer.  The  maintainers  stated  that  for  many  novices, 
mental  rotation  is  a  skill  not  easily  learned.  One  maintainer  contradicted  the  other  two  and  stated 
that  he  believed  all  the  graphics  are  in  a  proper  perspective.  The  final  maintainer  did  not  respond 
to  the  question. 

ABDR  Maintenance:  For  the  majority  of  the  maintainers,  ABDR  graphics  required  the 
same  level  of  mental  rotation  as  routine  maintenance  graphics.  Again,  the  wiring  systems  and 
wing  portions  of  the  aircraft  required  the  most  frequent  mental  rotation.  Two  maintainers 
contradicted  each  other.  One  stated  that  ABDR  repairs  are  more  general  than  routine  repairs,  thus 
requiring  less  mental  rotation,  while  the  other  stated  that  ABDR  repairs  require  more  mental 
rotation. 

Question:  Was  there  ever  a  time  when  you  had  to  change  your  visualization  of  a  TQ  graphic 
to  complete  a  job?  When? 

Routine  Maintenance:  All  but  one  of  the  maintainers  answered  this  question 
affirmatively.  Maintainers  approach  repairs  with  a  preconceived  notion  of  how  to  start.  The  TO 
graphics  are  referenced  to  complete  the  plan  of  action.  However,  the  repairs  are  often  not 
visually  consistent.  That  is,  hydraulic  lines,  wires,  bullets,  and  wing  structures  may  not  be 
oriented  like  the  graphic.  Once  this  realization  is  made,  the  maintainer  can  either  change  his 
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mental  visualization  of  the  graphic  to  coincide  with  the  plane  or  re-orient  himself  to  be  consistent 
with  the  visualization.  To  aid  in  the  former,  some  maintainers  will  physically  re-orient  the 
graphics.  Of  course,  any  associated  text  is  now  in  a  non-standard  orientation  and  thus  much  less 
readable.  The  goal  of  this  procedure  is  to  generate  graphics  visually  consistent  with  the  repair. 

ABDR  Maintenance:  Of  the  three  maintainers  who  responded,  all  claimed  to  have 
changed  their  visualization  of  a  graphic  to  complete  a  job.  The  justifications  for  doing  this  were 
identical  to  the  ones  given  when  asked  the  same  question  with  regard  to  routine  maintenance. 

Question:  Was  there  ever  a  time  when  graphics  were  unavailable?  When? 

Routine  Maintenance:  The  majority  of  maintainers  responded  that  there  was  never  a  time 
that  graphics  were  unavailable.  To  the  best  of  their  recollection,  they  were  never  in  need  of  a 
graphic.  Two  of  the  six  maintainers  recalled,  however,  a  need  for  an  apparently  missing  graphic. 
The  repair  areas  included  the  wing  and  environmental  duct.  There  were  no  graphics  available  to 
aid  them  in  removing  and  replacing  these  items. 

ABDR  Maintenance:  All  but  one  of  the  five  maintainers  responded  that  there  was  never  a 
need  for  additional  graphics.  The  one  requirement  for  graphics  was  to  aid  in  the  search  for 
projectiles  (objects  which  have  pierced  the  aircraft’s  shell). 

Question:  Would  3D  graphics  aid  in  typical/ABDR  maintenance?  When? 

Routine  Maintenance:  All  six  of  the  maintainers  gave  an  overwhelmingly  positive 
response  to  the  potential  usefulness  of  3D  graphics.  They  stated  that  this  technology  would  allow 
them  to  have  more  control  over  the  graphics.  For  example,  the  graphics  could  be  manipulated  to 
produce  the  same  perspective  as  that  of  the  repair  area  on  the  aircraft.  The  graphics  could  also 
contain  more  information  and  realism.  Providing  different  sets  of  graphics  for  different  types  of 
jobs  was  another  suggestion.  The  level  of  detail  in  the  graphic  would  depend  on  the  job  it  aided. 
Overall,  the  maintainers  felt  that  3D  graphics  would  better  help  them  identify  and  understand 

repairs.  ■; 


ABDR  Maintenance:  All  but  one  of  the  maintainers  believed  ABDR  maintenance  would 
benefit  from  3D  graphics.  They  could  see  that  the  use  of  3D  graphics  would  help  in  the 
assessment  phase,  specifically  damage  assessment.  With  the  help  of  3D  graphics,  the  maintainers 


could  locate  projectiles  and  the  extent  of  damage  by  better  understanding  the  plane’s  structure 
provided  through  3D  graphics.  Some  tasks  have  to  be  performed  without  visual  feedback  due  to 
limited  accessibility  compartments,  and  3D  graphics  can  provide  this  otherwise  missing 
information  (features  not  visible).  The  identification,  location,  and  path  of  wires  could  also  be 
conveyed  within  a  3D  graphic.  The  only  maintainer  who  did  not  agree  that  3D  graphics  would 
aid  in  ABDR  maintenance  was  an  electrician.  He  contradicted  another  maintainer  by  saying  that 
wiring  repairs  would  not  benefit  from  this  technology.  He  believed  graphics  represent  parts  that 
have  not  been  damaged  and  that  the  only  way  to  properly  assess  and  repair  parts  is  to  go  directly 
to  the  aircraft.  He  saw  no  benefit  in  having  3D  graphics. 

Question:  Would  3D  graphics  aid  a  novice?  When? 

Routine  Maintenance:  Of  the  four  maintainers  who  responded  to  this  question,  all  felt  that 
3D  graphics  would  aid  a  novice.  One  maintainer  stated  that  the  current  graphics  were  not  that 
helpful  to  a  novice  and  that  3D  graphics  would  provide  more  details,  making  the  jobs  easier  to 
perform.  The  more  detailed  the  graphic,  the  more  helpful  they  will  be  to  a  novice.  Another 
maintainer  felt  that  the  use  of  3D  graphics  in  training  would  cut  down  on  training  time 
tremendously.  Although  this  last  point  has  not  been  validated,  the  use  of  3D  graphics  in  training 
warrants  further  investigation. 

ABDR  Maintenance:  Again  only  four  maintainers  responded  to  this  question  from  the 
perspective  of  ABDR.  Just  as  before,  they  all  stated  that  3D  graphics  would  help  novices.  The 
added  capabilities  3D  graphics  provide,  such  as  enlarging  and  labeling  graphics,  could  only 
benefit  a  novice.  Because  there  are  so  many  voids  and  crevices  in  the  aircraft,  maintainers  have  a 
difficult  time  identifying  and  accessing  particular  areas.  3D  graphics  offers  a  method  of 
“accessing”  a  concealed  area.  This  feature  could  prove  to  be  extremely  helpful.  The  ability  to 
change  the  perspective  of  a  graphic  was  also  desirable. 

Question:  Which  TQ  graphics  need  improvement? 

Routine  Maintenance:  All  six  of  the  maintainers  responded  to  this  question.  Graphics 
showing  component  removal,  wiring  and  hydraulic  diagrams,  wings,  and  flight  controls  need 
improvement  to  produce  a  more  accurate  representation  of  the  aircraft.  These  graphics  were 
selected  because  of  specific  deficiencies,  but  there  was  a  common  trend  among  them.  The 
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majority  of  the  complaints  centered  on  the  poor  perspective  of  the  graphics  and  the  difficulty  of 
extracting  information.  Often,  maintainers  cannot  even  locate  the  appropriate  graphic.  As  stated 
many  times,  this  is  the  result  of  the  TO  manual’s  poor  layout.  Their  organization  was  criticized 
as  being  confusing  and  inconsistent. 

ABDR  Maintenance:  Fewer  maintainers  responded  to  the  same  question  when  asked  with 
regard  to  ABDR  maintenance.  Again,  the  wing  graphics  were  chosen  as  needing  improvement.  A 
maintainer  stated  that  these  graphics  were  cluttered  and  often  presented  from  unusual 
perspectives.  Another  design,  side-by-side  graphics  showing  opposite  views,  also  requires  a 
better  perspective  or  labeling  to  make  orientation  information  clearer  with  respect  to  the  aircraft. 
For  example,  this  scenario  occurs  in  graphics  of  the  fuselage.  One  maintainer  stated  that  the  lack 
of  details  found  in  the  current  graphics  make  ABDR  maintenance  tasks  even  more  difficult. 

Question;  What  type  of  improvement  is  needed  for  the  TO  graphics? 

Routine  Maintenance;  All  six  of  the  maintainers  answered  this  question.  There  were  a 
wide  variety  of  responses.  One  called  for  graphics  to  appear  in  the  same  perspective  as  on  the 
aircraft.  The  demand  for  the  ability  to  zoom,  peel  away  or  remove,  rotate,  show  color  and  label 
components  was  repeated  by  virtually  every  maintainer.  There  was  a  stipulation  placed  on  the 
color  feature,  in  that,  if  used,  colors  must  closely  resemble  the  colors  found  on  the  aircraft.  The 
capability  to  show  a  wiring  harness  or  a  hydraulic  line  from  start  to  finish  was  a  desired 
improvement.  Along  with  the  capability  of  labeling  components,  additional  information  should 
be  provided,  such  as  the  structural  makeup,  sub-components,  and  dimensions.  Several  unique 
features  also  surfaced,  for  example,  a  search  feature  that  would  allow  the  maintainers  to  locate  a 
part  or  graphic  instantly.  The  ability  to  simulate  damage  in  the  3D  graphic  so  an  engineer  could 
see  the  damage  rather  than  just  rely  on  the  maintainer's  description  could  lead  to  a  more  efficient 
repair  process. 

ABDR  Maintenance:  The  six  maintainers  had  similar  requests  in  response  to  this  question 
regarding  ABDR.  The  most  popular  suggestion  for  improvement  was  a  feature  not  considered  by 
the  research  team.  All  of  the  maintainers  called  for  some  type  of  tutorial  or  help  system  that 
would  offer  hints  for  difficult  repairs  or  store  experts’  solutions  to  uncommon  repairs.  The  hints 
would  provide  help  regarding  repair  procedures  for  a  particular  area.  These  hints  would  be 
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gathered  through  experience,  stored  in  the  computer  along  with  3D  graphics  and  relayed  to  the 
maintainer  performing  the  repair.  One  maintainer  suggested  that  damage  be  assessed  and  typed 
into  a  computer,  which  would  recommend  an  ABDR  repair  procedure  (artificial  intelligence). 

Question:  Do  you  ever  visualize  peeling  away  levels  of  surface?  When? 

Routine  Maintenance:  Four  of  the  six  maintainers  responded  yes  to  this  question.  All  had 
either  performed  a  mental  visualization  effectively  peeling  away  layers  of  the  aircraft,  such  as 
panels,  wires  and  wing  surfaces,  and  thought  that  a  3D  graphic  would  take  the  place  of  having  to 
perform  this  function  mentally. 

ABDR  Maintenance:  Again,  only  four  responses  were  recorded  for  this  question.  Peeling 
away  the  skin  of  the  aircraft  and  the  removal  of  components  were  the  most  typical  scenarios 
reported. 

Question:  Is  full  manipulation  of  3D  graphics  required? 

Routine  Maintenance:  A  little  over  half  of  the  maintainers  answered  this  final  question. 
Three  felt  that  full  manipulation  capabilities  of  the  3D  graphic  are  required  to  use  3D  graphics  to 
their  fullest  extent.  The  other  maintainer  stated  that  once  a  graphic  gets  down  to  a  specific  level 
of  detail,  only  three  views  would  be  needed  to  complete  a  job. 

ABDR  Maintenance:  None  of  the  maintainers  responded  to  this  question  when  asked  in 
the  ABDR  session. 


Capability  Requirements 

The  six  maintainers  who  participated  in  the  CTA  provided  numerous  comments  and 
suggestions  as  to  the  use  of  current  TO  graphics  and  the  potential  use  of  3D  graphies  in 
maintenance  tasks.  All  of  the  maintainers  stated  that  the  current  TO  graphics  do  aid  in 
performing  tasks  but  could  use  some  improvement.  The  overall  consensus  was  that  3D  graphics 
would  provide  an  added  benefit.  A  list  of  requirements  that  were  deemed  essential  to  the  use  of 
3D  graphics  has  been  culled  from  the  maintainer’ s  comments. 


31 


1 .  Full  manipulation  of  a  3D  graphic 

This  feature  would  allow  users  to  manipulate  the  graphic  in  any  manner  that 
suited  the  repair  purpose. 

2.  Labeling  of  components 

This  feature  would  help  in  the  identification  and  location  of  components  within  a 
graphic. 

3.  Removal  of  components  capability 

This  feature  would  allow  components  to  be  removed  so  that  items  located  behind 
them  could  be  viewed.  This  would  be  particularly  helpful  in  the  assessment  phase. 


4.  Zoom 


The  zoom  feature  would  allow  users  to  get  a  closer  view  of  a  particular 
component.  However,  this  feature  is  useful  only  if  the  closer  view  provides 
increased  detail. 


5.  Rotate 

This  feature  would  give  the  maintainers  the  ability  to  view  a  component  in  the 
best  possible  position. 

6.  Display  of  colors 

The  use  of  color  with  3D  graphics  is  beneficial.  The  components  can  either  be 
color-coded  or  colored,  as  realistically  as  possible,  as  they  appear  on  the  aircraft. 

The  maintainers  also  called  for  a  variety  of  other  features  that  would  aid  both  a  novice 
and  expert  in  their  maintenance  task.  Several  of  the  maintainers  suggested  that  separate  sets  of 
graphics  pertaining  to  different  types  of  jobs  would  be  an  efficient  way  to  organize  the  TO 
information.  A  “wide-angle”  view  would  also  allow  maintainers  to  see  components  in  an 
electronic  format  that  would  typically  be  missing  from  a  current  TO  graphic.  Furthermore,  the 
maintainers  suggested  that  more  details  than  those  provided  by  current  TO  graphics  are  needed 
to  further  aid  in  maintenance  procedures. 

Finally,  scale  drawings  along  with  the  ability  to  show  damage  on  the  3D  graphic  would 
allow  the  maintainers  to  draw  damage  on  a  graphic  just  as  it  exists  on  the  aircraft.  This  would 
give  the  engineer  a  better  idea  of  the  extent  of  the  damage. 
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Another  feature  greatly  desired  by  the  maintainers  was  the  ability  to  search  a  TO  for  a 
particular  component  by  typing  the  name  and/or  part  number  into  the  computer  and  having  that 
part  appear  highlighted  in  the  3D  graphic.  They  also  mentioned  several  times  the  desire  to  type 
in  the  location  and  extent  of  the  damage  and  have  the  computer  recommend  repairs  or  provide 
hints  and  tips  of  things  to  do.  These  hints  or  tips  would  definitely  prove  useful  for  both  the 
novice  and  experienced  maintainers.  For  example,  ABDR  maintainers  are  often  qualified  for  a 
variety  of  system  repairs  but  may  typically  only  work  on  a  few  particular  systems.  As  a  result, 
there  may  be  systems  and  system  repairs  with  which  the  maintainer  may  be  less  familiar.  Hints 
to  guide  the  appropriate  repair  procedures  would  be  useful. 

Finally,  several  maintainers  commented  that  3D  graphics  would  cut  down  on  training 
time  tremendously.  Although  a  supplemental  study  would  be  needed  to  determine  if  this  claim 
could  be  validated,  the  maintainers’  emphatic  belief  indicated  that  further  study  is  warranted. 
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3D  FILE  FORMATS 


Review  of  Current  3D  File  Formats 

During  this  part  of  the  study,  the  team  identified  and  ranked  industry  and  DoD  3D  file 
formats.  In  order  to  rank  the  file  formats,  a  set  of  weighted  criteria  were  developed  and  used  to 
give  each  of  the  file  formats  a  numerical  ranking.  Details  of  the  criteria  and  ranking  results  are 
provided  in  the  following  sections.  The  team  determined  that  there  are  over  150+  common  3D 
graphics  file  formats  available,  each  with  their  own  set  of  features  and  characteristics. 

The  team  was  faced  with  one  of  three  possible  choices  for  identifying  and  evaluating  the 
industry  and  DoD  file  format  standards: 

■  Evaluate  all  of  the  150+  file  formats  through  the  entire  set  of  evaluation  criteria. 

■  Pass  all  of  the  150+  file  formats  through  a  small  subset  of  the  evaluation  criteria,  focusing  on 
the  criterion  with  a  “high”  importance  value. 

"  Leverage  the  extensive  knowledge  of  virtual  reality  (VR)  and  computer  graphics  systems  the 
team  possessed  to  logically  reduce  the  number  of  available  file  formats  to  a  workable  subset 
in  line  with  the  scope  of  this  effort. 

The  first  choice,  pass  all  file  formats  through  the  full  set  of  criteria,  was  immediately 
discarded  due  to  the  limited  amount  of  time  and  money  allotted  to  the  effort.  Evaluating  the  150+ 
file  formats  could  be  considered  a  never-ending  investigation  of  its  own,  since  the  file  formats 
are  constantly  being  updated,  and  new  ones  developed. 

The  second  choice,  while  appealing,  was  also  discarded  due  to  time  constraints  as  well  as 
the  focus  of  the  project:  the  3D  viewer.  The  file  formats  chosen  for  the  investigation  do  have 
some  merit  and  are  important  because  they  contain  a  set  of  data  the  viewer  will  use.  However, 
the  viewer  typically  dictates  the  file  format  as  discussed  in  the  section  titled  “3D  VIEWERS.” 
Therefore,  the  decision  was  made  to  focus  more  on  identifying  a  larger  number  of  viewers  at  the 
expense  of  evaluating  a  larger  number  of  3D  file  formats. 
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The  team  decided  on  the  third  choice,  to  rely  on  the  team’s  extensive  collective 
knowledge  of  the  computer  graphics  and  simulation  arena.  The  five  team  members  who  worked 
on  identifying  and  evaluating  the  file  formats  collectively  had  over  20  years  direct  experience  in 
computer  graphics.  Additionally,  members  of  the  AFRL/HES  division  with  computer  graphics 
experience  were  solicited  to  help  select  the  list  of  file  formats  that  were  finally  evaluated. 

Identification  of  3D  File  Formats 


The  file  formats  finally  chosen  by  the  team  had  one  or  more  of  the  following  features. 
One,  the  format  had  some  history  of  extensive  use  in  the  field  of  VR  and  3D  modeling.  Two,  the 
format  was  created  by  a  software  package  with  a  history  of  extensive  use  in  the  field  of  VR  and 
3D  modeling.  Three,  the  format  promoted  itself  as  a  format  with  open  standards.  Additionally, 
the  format  was  chosen  if  it  was  recommended  by  a  military  branch  concerned  with  VR  or 
computer  graphics. 

Table  1  lists  the  industry  standard  3D  file  formats  as  determined  by  the  research  team. 
The  first  column  lists  the  name  of  the  file.  The  second  column  lists  the  common  file  extensions 
associated  with  the  particular  file  format.  Finally,  the  last  column  identifies  the  company  or 
individual  responsible  for  the  development  and  support  of  the  file  format. 


Table  1.  Industry  Standard  3D  File  Formats 


Format  Name 

File  ExtensionCs) 

3DS  (3D  Studio) 

3ds 

Kinetix/  AutoDesk,  Inc. 

DWB  (Designers  Workbench) 

.dwb,  .dwba 

Coryphaeus  Software 

DXF  (Drawing  Interchange  File) 

.dxf 

AutoDesk  Inc. 

FLT  (OpenFlight) 

.fit 

Multigen 

NFF/  ENNF  (Neutral  File  Format/  Extended 
Neutral  File  Format) 

.nff,  .enff 

Eric  Haines 

OBJ  (Wavefront  Object) 

.obi,  .mod 

Wavefront/  Alias  Technologies 

POV  (Persistence  Of  Vision) 

^22:: _ 

POV“Ray  Team 

VRML  (Virtual  Reality  Modeling 

Language) 

.vrml,  .wrl 

Standard  based  upon  SGTs 

Open  Inventor  format 

In  addition  to  the  seven  commercially-produced  3D  file  formats,  the  team  identified  two 
potentially  viable  DoD  3D  file  formats.  Table  2  summarizes  the  DoD  file  formats  identified. 
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Table  2.  DOD  3D  File  Formats 


Format  Name 

File  Extension(s) 

SEDRIS  (Synthetic  Environment  Data 
Representation  &  Interchange  Specification) 

None,  format  is  still 
being  defined 

Defense  Modeling  and 

Simulation  Office  (DMSO) 

IGES  (Initial  Graphics  Exchange 
Specification) 

•jgs 

IGES/PDES  Organization 

Evaluation  Criteria  for  3D  File  Formats 

A  set  of  weighted  criteria  was  developed  to  evaluate  and  rank  each  of  the  file  formats 
identified.  This  list  of  criteria,  which  encompasses  many  uses  and  capabilities,  was  created  by 
focusing  on  the  following  needs  which  were  applicable  to  the  aircraft  maintenance  task: 

1)  The  format  has  some  feature  which  enhances  realism  of  the  simulated  environment. 

2)  The  format  has  some  feature  which  allows  for  manipulation  of  the  simulated 
environment. 

3)  The  format  has  some  feature  which  enhances  its  use,  its  distribution,  or  its  ability  to  be 

modified. 

As  in  the  selection  of  file  formats,  the  list  of  evaluation  criteria  was  distributed  to 
members  of  AFRL/HES  division  for  comment.  Table  3  summarizes  the  criteria  used  to  evaluate 
and  rank  each  of  the  file  formats. 

For  each  of  the  criterion  listed  in  the  first  column  of  Table  3,  there  is  an  associated  weight 
and  a  set  of  sub-criteria.  The  weight  associated  with  each  criterion  signifies  the  relative 
importance  of  that  particular  criterion.  Each  of  the  criterion  was  assigned  an  integer  weight  value 
between  10  and  1,10  being  the  highest  (most  important)  and  1  being  the  lowest  (least  important.) 
These  weights  were  determined  by  features  requested  by  HESR  representatives,  requests  from 
the  group  of  maintenance  personnel  during  the  CTA,  and  the  need  to  produce  a  realistic, 
modifiable  simulated  environment  that  could  be  manipulated. 

For  example,  the  criteria  “Text”  was  given  a  weight  of  8  because  textural  information 
was  highly  recommended  by  the  customer.  “Groupings”  was  given  a  9  because  the  capability  to 
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represent  complex  objects  was  highly  prized  for  realistic  simulations.  “Color”  was  given  a  7 
because  several  maintenance  personnel  liked  the  idea  of  marking  items  in  color.  On  the  other 
side  of  the  list,  items  such  as  “Wireframe”  were  given  low  scores  because  those  items  were  not 
mentioned  by  the  customer  or  maintenance  personnel  and  do  not  lend  themselves  well  to 
producing  the  most  realistic  environments. 

To  be  as  objective  as  possible,  each  criterion  was  broken  down  into  a  series  of  sub  - 
criteria  which  represented  capabilities  that  the  main  criteria  can  contain.  Each  sub-criterion, 
shown  in  the  third  column  of  Table  3,  is  separated  with  a  square  box.  For  example,  the  criterion 
“Polygon”  consists  of  four  sub  criteria:  “Convex,  N-sided,  Concave,  and  Decaled.”  The 
numerical  value  listed  prior  to  each  sub-criteria  is  the  specific  score  that  a  particular  sub-criterion 
contributes  to  the  overall  score  of  the  criterion.  The  sum  of  the  numerical  values  associated  with 
each  of  the  sub-criteria  always  adds  to  a  value  of  10.  The  score  a  file  format  received  for  a 
particular  criterion  was  calculated  as  follows. 

1)  Determine  which  of  the  sub-criteria  were  fulfilled  by  the  file  format 

2)  Add  the  numerical  values  of  the  satisfied  sub-criteria,  a  value  between  0-10. 

3)  Take  the  value  obtained  from  step  2  and  multiply  by  the  weight  factor 

For  example,  assume  that  file  format  X  supported  convex  and  N-sided  polygons  but  lacked 
support  for  concave  and  decaled  polygons.  File  format  X  would  have  received  a  score  of  80 
[(6+2)  *  10]  for  the  criterion  polygons.  For  the  hypothetical  file  format  X,  80  is  considered  the 
“raw  score”  for  this  particular  criterion. 

While  realism  and  manipulation  play  a  much  larger  part  in  the  3D  viewer  criteria,  they 
also  play  a  part  in  the  file  format  ranking.  The  file  format,  which  contains  the  sets  of  data,  allows 
realism  and  manipulation  of  the  environment  in  the  3D  viewer. 
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Table  3.  3D  File  Format  Evaluation  Criteria 


Criterion 

wt 

Sub-Criteria  - ^ — _ 

Polygons 

10 

□  (6)  Convex:  Polygons  stored  can  have  a  convex  shape. 

□  (2)  N-sided:  Polygons  stored  can  have  an  unlimited  number  of  sides. 

□  (})  Concave:  Polygons  stored  can  have  a  concave  shape. 

□  (1)  Decaled:  Polygons  can  be  stored  one  within  another  on  the  same  plane. 

Render  order  information  is  included,  allowing  the  polygons  to  be  rendered 
properly.  _ 

Groupings 
and  Links 

9 

□  (6)  Grouping:  The  stored  information  represents  that  one  or  more  polygons  can  be 
considered  as  a  logical  group  with  a  common  transformation  matrix. 

□  (3 )  Links  to  external  files:  There  can  be  links  to  files  of  the  same  format,  which 
impart  additional  grouping  information. 

□  (1 )  Named  groupings:  Each  grouping  has  a  descriptive  name. 

Lines  and 
Lines  styles 

8 

□  (7)  Lines:  Lines  are  stored  in  the  format  as  a  primitive. 

□  (2)  Patterns:  Information  associated  with  the  line  primitive  pattern  is  stored. 

□  (1 )  Thickness:  Information  associated  with  the  line  thickness  is  stored. 

Text 

8 

□  (5)  Primitive:  Text  can  be  stored  as  a  primitive. 

□  (2)  Size:  Information  associated  with  the  text’s  size  is  stored. 

□  (1)  Font:  Information  associated  with  the  text’s  font  is  stored. 

□  (1)  Direction:  Information  associated  with  the  direction  the  text  is  oriented  is 
stored. 

□  (1 )  Justification:  Information  associated  with  the  text’s  justification  is  stored. 

External 
links  to  text 

8 

□  (10)  Text:  Links  to  files,  which  contain  text  information,  are  stored. 

External 
links  to 
pictures 

8 

□  (JO)  Pictures:  Links  to  files  which  contain  2D  bitmap  information  are  stored. 

_ - — - - 

Mathematical 

Objects 

7 

□  (3)  Common  3D  objects:  Spheres  or  boxes  can  be  stored  as  primitives. 

□  (2)  Common  2D  objects:  Ellipses  or  n-sided  figures  can  be  stored  as  primitives. 

□  (2)  Less  common  3D  objects:  Torii,  or  discs,  or  cylinders,  or  cones,  or  surfaces  of 
revolution  can  be  stored  as  primitives. 

□  (1)  Less  common  2D  objects:  Splines  or  arcs  can  be  stored  as  primitives. 

□  (1)  Rare  3D  objects:  Fractals  or  mathematical  surfaces  can  be  stored  as  primitives. 

□  (1)  Constructive  Solid  Geometry:  Logical  operations  (such  as  AND,  OR)  are 

stored  in  the  format  for  the  purpose  of  creating  a  new  object  out  of  a  set  of 
mathematical  objects. 

Color 

Information 

7 

□  (7)  Unlimited  colors:  R,  G,  B  values  or  an  unlimited  number  of  color  table  values 
are  stored. 

□  (3 )  Limited  colors:  A  limited  number  of  color  table  values  are  stored. 

Transparency 

7 

□  ( 10)  Transparency:  A  transparency  value  is  stored  with  a  primitive. 

Material 

Properties 

6 

□  (2 )  Specular:  Specular  lighting  components  are  associated  with  a  primitive. 

□  (2)  Diffuse:  Diffuse  lighting  components  are  associated  with  a  primitive. 

□  (2)  Ambient:  Ambient  lighting  components  are  associated  with  a  primitive. 

□  (I)  Emissive:  Emissive  lighting  components  are  associated  with  a  primitive. 

□  (1)  Shininess:  Shininess  lighting  components  are  associated  with  a  primitive. 

□  (1)2D  Bump  mapping:  A  2D  set  of  data  representing  a  bump  map  is  associated 
with  a  primitive. 

□  (1)  3D  Bump  mapping:  A  3D  set  of  data  representing  a  bump  map  is  associated 
with  a  primitive. 
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Grit^ribii 

mi:. 

Sub'Criteria 

Textures 

6 

□  (6)  2D  Texture  mapping-.  A  2D  set  of  data  representing  a  texture  map  is  associated 
with  a  primitive. 

□  (3 )  Draw  style:  A  draw  style  representing  a  particular  repetitive  pattern  is 
associated  with  a  primitive. 

□  (1)  3D  Texture  mapping:  A  3D  set  of  data  representing  a  texture  map  is  associated 
with  a  primitive. 

Normals 

5 

□  (8)  Vertex  normals:  Normal  vectors  are  stored  and  associated  with  a  primitive’s 
vertex, 

□  (2)  Face  normals:  Normal  vectors  are  stored  and  associated  with  a  primitive’s 
face. 

Viewing 

Information 

5 

□  (4)  Multiple  views:  More  than  one  specified  view  angle  and  position  are  stored. 

□  (3)  One  view:  A  single  specified  view  angle  and  position  is  stored. 

□  (3)  Fly-though:  More  than  one  specified  view  angle  and  position  is  stored,  and  a 
pattern  representing  a  fly-though  sequence  is  stored. 

Efficiency 

Patterns 

5 

□  (2)  Polygon:  Polygons  can  be  stored  in  strips,  fans,  and  meshes. 

□  (2 )  Vertex  list:  Vertices  are  stored  in  a  list. 

□  (2)  Tables:  Color  tables  and  material  tables  are  supported, 

□  (2)  Global  lists:  Vertex  lists,  or  texture  lists,  or  normal  lists  are  considered  global 
and  allow  other  groupings  to  access  them. 

□  (I)  Texture  vertex  list:  Texture  coordinates  are  stored  in  a  list. 

□  (I )  Normal  list:  Normal  vectors  are  stored  in  a  list. 

Render  Order 

4 

□  (7)  Strict  order:  Values  associated  with  the  rendering  order  of  primitives  are 
stored.  The  values  are  in  a  format,  which  indicates  the  exact  order  that  all 
primitives  will  be  rendered  in.  It  is  possible  to  have  strict  order  without  having 
the  capability  for  loose  order. 

□  (3)  Loose  order.  Values  associated  with  the  rendering  order  of  primitives  are 
stored.  The  values  are  in  a  format  which  indicates  a  rough  order  (front,  back,  or 
grouping  levels)  that  all  primitives  will  be  rendered. 

Openness 
and  Support 

4 

! 

□  (5)  Governing  group:  The  group,  which  dictates  features,  changes,  additions,  and 
implementation  of  the  format  is  a  fairly  impartial  one. 

□  (4)  Support:  The  group,  which  governs  the  file  format  and  a  subset  of  users,  offer 
freely  available  support. 

□  (J)  Specifications:  The  group,  which  governs  the  file  format,  allows  the  full  file 
format  specifications  to  be  known  and  freely  distributed  with  no  legal  bindings. 

Measurement 

Units 

3 

□  (4)  Linear  per  primitive:  English  and  metric  linear  units  are  associated  with  a 
primitive. 

□  (3)  Angular  per  primitive:  Radian  and  degree  angular  units  are  associated  with  a 
primitive. 

□  (2)  Linear  per  file:  English  and  metric  linear  units  of  measure  are  associated  with 
all  primitives  in  the  file. 

□  (1)  Angular  per  file:  Radian  and  degree  angular  units  of  measure  are  associated 

with  all  primitives  in  the  file.  . 

Points 

3 

□  (9)  Points:  Points  are  stored  in  the  format  as  a  primitive. 

□  (1)  Size:  Information  associated  with  the  point  size  is  stored. 

Switches, 

LOD, 

Morphing 

3 

□  (4)  Switches:  Several  versions  of  a  primitive’s  geometry  are  stored  in  the  format. 

□  (3 )  Named  States:  Each  state  has  a  descriptive  name. 

□  (2)  LOD:  Distance  information  is  stored  with  the  versions  to  indicate  when  to 
switch  between  states. 

□  (1)  Morphing:  The  versions  are  marked  in  some  way  to  allow  a  smooth  transition 
between  the  versions  of  the  geometry. 
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Criterion 

■wt 

Sub-Criteria  . 

Bounding/ 

Culling 

Information 

3 

□  (3)  Complex  boundaries:  Information  describing  a  bounding  area  of  a  complex 
shape  around  the  primitive’s  geometry  is  stored. 

□  (7)  Mathematical  object  boundaries:  Information  describing  a  bounding  area  of  a 
simple  2D  or  3D  mathematical  object  shape  around  the  primitive’s  geometry  is 
stored. 

User 

Extensible 

3 

□  (4)  Customization-.  The  file  format  allows  the  definition  of  sets  of  data,  the 
function  of  which  is  not  defined  in  the  file  format  specifications.  File  formats, 
which  extend  themselves  in  this  manner  through  comment  fields,  are  excluded. 

□  (4)  Code  Sets:  The  file  format  can  contain  some  type  of  computer  language. 

□  (2)  Field  extensibility:  The  file  format  contains  some  fields  which  the  user  can 
store  information. 

Wireframe 

3 

□  (7)  Wireframe:  Any  of  the  primitives  elsewhere  in  the  criteria  list  can  be  stored 
specifically  as  a  wireframe. 

□  (2)  Patterns:  Information  associated  with  the  pattern  of  the  wireframe  is  stored. 

□  (1 )  Thickness:  Information  associated  with  the  thickness  of  the  wireframe  is 
stored. 

Animations/ 

Articulations 

2 

□  (6)  3D  Animation:  An  animation  sequence  created  by  changing  transformation 
parameters  of  primitives  is  stored. 

□  (3)  3D  Articulation:  Parameters  representing  limits  to  degrees  of  freedom  for 
primitives  are  stored. 

□  (I)  2D  Animation:  An  animation  sequence  created  by  sequencing  through  texture 
maps  or  material  property  patterns  on  a  primitive’s  surface  is  stored. 

Space 

Partitioning 

2 

□  (W)  Partitioning:  Any  form  of  space  partitioning  information  is  stored. 

Light 

Sources 

2 

□  (4)  Infinite  light  source:  Infinite  light  source  parameters  are  stored. 

□  (4)  Local  light  source:  Local  light  source  parameters  are  stored. 

□  (1)  Spotlight  light  source:  Spotlight  light  sources  parameters  are  stored. 

a  (1)  Color:  RGB  values  or  color  table  values  can  be  associated  with  a  light  source. 

Stability  of 
Format 

2 

□  (8)  Compatibility:  All  new  formats  are  backward  compatible  with  previous 
versions  of  the  format. 

□  (2)  Updates:  The  governing  group  changes  the  format  specifications  less  than 
once  a  year. 

Size  of  3D 
Files 

2 

□  (10)  Size:  The  size,  in  kilobytes,  of  several  task-related  objects  stored  in  the 

format. 

Links  to 
sound  files 

2 

□  (10)  Sound:  Links  to  external  files,  which  contain  digitized  sound  information,  are 

supported. 

Comments 

1 

□  (7)  Nonrestrictive  comments:  There  exists  some  comment  marker,  which  allows 
the  inclusion  of  comments  at  any  position  in  the  file. 

□  (2)  Restrictive  comments:  Comments  can  be  placed  in  certain  fields  in  the 

structure:  _ 

3D  File  Format  Evaluation  Results 

This  section  contains  the  results  of  evaluating  each  of  the  identified  file  formats  using  the 
evaluation  criteria  listed  in  Table  3.  Table  4  summarizes  how  each  of  the  file  formats  performed 
in  the  evaluation.  The  file  formats  are  listed  across  the  top  of  the  table  starting  in  the  second 
column.  The  first  column  of  the  table  identifies  the  criterion.  The  numerical  values  for  each  of 
the  file  format  evaluation  criterion  combinations  represents  the  raw  score.  The  raw  score  is 
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defined  as  the  sum  of  the  satisfied  sub-criteria  times  the  weight  for  the  particular  criterion.  The 
last  two  rows  of  Table  4  list  the  raw  score  total  and  the  normalized  score  for  each  of  the  file 
formats.  The  raw  score  total  was  calculated  by  adding  all  of  the  raw  scores  for  each  of  the 
criteria.  Dividing  the  total  raw  score  by  the  total  possible  points,  in  this  case  1260,  produced  a 
normalized  score  for  each  of  the  file  formats  (a  score  between  0-100%,  where  100%  indicates  the 
file  format  met  all  the  criteria). 


Table  4.  3D  File  Format  Evaluation  Results 


VRML 

SEDRIS 

DWB 

FLT 

FOV 

IGES 

3PS 

DXF 

,  om 

NEF/ 

ENNF^ 

Polygons 

100 

100 

100 

100 

90 

90 

60 

70 

90 

90 

Groupings  and  Links 

90 

90 

63 

90 

90 

63 

90 

0 

Lines  and  Unestyles 

56 

56 

0 

0 

0 

80 

0 

80 

56 

0 

Text 

80 

48 

80 

64 

72 

80 

0 

80 

0 

0 

External  Links  to  Text 

80 

0 

0 

0 

0 

0 

0 

0 

0 

0 

External  Links  to  Pictures 

80 

80 

80 

0 

0 

0 

80 

0 

0 

0 

Mathematical  Objects 

35 

42 

7 

49 

14 

49 

Color  Information 

70 

70 

70 

21 

70 

70 

70 

0 

70 

■  70 

70 

70 

70 

70 

0 

70 

70 

0 

Material  Properties 

48 

54 

48 

48 

48 

0 

54 

0 

36 

42 

Textures 

54 

54 

54 

54 

54 

18 

36 

18 

60 

0 

Normals 

50 

50 

40 

HIEMI 

0 

0 

40 

50 

Viewing  Information 

50 

35 

35 

15 

50 

15 

0 

15 

20 

50 

10 

40 

10 

25 

10 

30 

0 

Render  Order  Information 

12 

40 

0 

12 

12 

40 

0 

0 

Openness  and  Support 

40 

40 

20 

0 

20 

20 

0 

20 

20 

20 

Measurement  Units 

0 

12 

6 

6 

0 

27 

0 

30 

0 

0 

Points 

27 

27 

30 

0 

30 

0 

30 

27 

0 

Switches,  LOD,  Morphing 

30 

30 

27 

30 

0 

3 

0 

0 

0 

Bounding  /  Culling  Info. 

30 

21 

21 

21 

30 

0 

21 

0 

0 

0 

User  Extensible 

30 

12 

18 

6 

0 

12 

0 

12 

0 

0 

Wireframe 

0 

0 

30 

30 

0 

0 

24 

0 

0 

0 

Animations  /  Articulations 

14 

20 

14 

20 

14 

0 

12 

0 

0 

0 

20 

20 

20 

0 

0 

0 

0 

0 

0 

Light  Sources 

20 

20 

20 

20 

12 

0 

iH9i 

0 

0 

10 

Stability  of  Format 

0 

0 

20 

4 

20 

20 

20 

20 

20 

Size  of  3D  Files 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

Links  to  Sound  Files 

20 

20 

20 

20 

0 

0 

0 

0 

0 

0 

Comment  Fields 

10 

3 

10 

10 

3 

0 

10 

10 

Raw  Score  Total 

IBiiMlM 

923 

-Mflgill 

563 

376 

1  82.3 

■  66.9 

1  56.2 

1  .  51.7 

i  :  4L6. 

.  40:8 

27.2 
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Descriptions  of  Identified  File  Formats 


This  section  provides  a  brief  description  of  each  evaluated  file  format,  including  details 
on  how  the  particular  file  format  performed  against  the  evaluation  criteria.  The  name  of  the  file 
format  is  presented  using  underlined  type  followed  by  the  description.  The  descriptions  are 
presented  in  ascending  order  based  on  the  overall  score  the  file  format  received  (see  Table  4). 

Virtual  Reality  Modeling  Language  (VRML),  82.3  % 

The  Virtual  Reality  Modeling  Language  is  an  attempt  to  define  an  open,  multi-platform, 
virtual  reality  language  for  the  Worldwide  Web  (WWW).  ASCII  files  containing  this  language 
are  used  in  conjunction  with  VRML  browsers,  allowing  a  Web  user  to  move  through  a  simulated 
environment. 

VRML  was  originally  developed  by  Silicon  Graphics,  Inc.,  which  used  its  Open  Inventor 
file  format  as  the  basis  for  VRML.  VRML  was  modified  to  be  Web  friendly  in  Version  1.0  of 
the  file  format  specification  released  on  May  26,  1995.  The  current  version  (2.0)  was  released 
August  4,  1996  by  the  VRML  Architecture  Group  (VAG).  The  VAG  is  a  collection  of 
companies  that  decide  what  and  when  changes  are  made  to  the  format  specification.  The 
specification  for  the  file  format  is  available  on  the  Internet  and  other  published  sources.  Support 
is  available  from  the  numerous  companies  that  comprise  the  VAG.  VRML  was  recognized  as  an 
ISO  and  lEC  standard  in  December  1997. 

VRML  is  popular  not  only  because  it  can  be  used  on  the  WWW,  but  because  it  has  many 
powerful  features,  including  the  specification  of  a  variety  of  different  kinds  of  individual  objects 
and  groups  of  objects;  text  specification;  material  and  color  compositions  and  lighting 
specification;  specification  of  interactive  behavior;  Level  Of  Detail  (LOD);  and  animation.  A 
feature  that  sets  VRML  apart  from  most  other  3D  file  formats  is  that  it  allows  users  to  extend  its 
capability. 

Although  VRML  is  a  powerful  tool  for  specifying  3D  scenes,  it  has  some  shortcomings. 
These  include  a  lack  of  support  for  measurement  units  and  complex  and  uncommon  2D  and  3D 
primitive  objects. 
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Synthetic  Environment  Data  Representation  &  Interchange  Specification  (SEDRIS),  77.1% 


The  SEDRIS  data  model  is  being  developed  by  several  government  agencies  (including 
the  Defense  Modeling  and  Simulation  Office)  as  well  as  13  private  simulation  and  government 
contractor  companies.  Although  the  SEDRIS  data  model  is  not  yet  fully  defined,  it  incorporates 
classes  of  data,  relationships  between  the  classes,  and  primitive  data  types  -  in  other  words,  it 
can  be  considered  a  language. 

Features  of  SEDRIS  encompass  nearly  all  the  criteria  required  for  this  investigation.  Its 
class  structure  gives  it  a  strong  groupings  and  links  capability,  and  most  of  the  basics  such  as 
polygons,  color  information,  and  transparency  are  included.  The  current  missing  features 
include  external  links  to  text  as  well  as  weak  support  for  measurement  units,  but  these  may 
eventually  be  included  in  the  final  revision. 

Designer’s  Workbench  (DWB).  66.9% 

The  Designer’s  Workbench  format  is  maintained  by  Coryphaeus  Software,  Inc.  for  use 
with  its  3D  Database  Modeler.  The  modeling  package.  Designer’s  Workbench,  creates  3D 
models  for  real-time  visual  simulation.  The  current  version  of  the  format  is  4.0,  which  was 
released  in  September  1997.  Files  are  in  a  binary  format  for  better  disk  space  utilization.  The 
physical  layout  is  designed  to  allow  any  unrecognized  entity  to  be  easily  ignored. 

The  conceptual  layout  is  hierarchical,  which  makes  this  format  a  little  more  difficult  to 
create  and  read,  but  adds  a  great  deal  of  power.  This  file  format  supports  most  of  the  basic  and 
higher  level  constructs  desired  in  a  3D  file  format.  It  has  very  good  support  for  polygons, 
groupings,  text,  colors,  transparency,  material  properties,  textures,  normals,  switches,  LOD, 
animations,  articulations,  and  light  sources.  Some  of  the  major  constructs  missing  from  the 
format  include  lines,  external  links,  mathematical  objects,  and  efficiency  patterns.  The  DWB 
format  contains  most,  if  not  all,  of  the  fields  needed  when  creating  models  for  real-time 
simulation  but  is  missing  some  of  the  fields  necessary  to  make  it  an  excellent  general-purpose 
3D  graphics  file  format. 
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OpenFlight  (rLT").  56.2% 

The  OpenFlight  3D  file  format  was  originally  developed  by  Multigen,  Inc.  for  use  in  their 
3D  modeling  and  real-time  graphics  software.  The  release  of  the  file  format  into  the  public 
domain,  coupled  with  the  versatility  and  features  of  the  format  itself,  have  made  OpenFlight  one 
of  the  most  widely-used  file  formats  in  the  real-time  simulation  and  computer  gaming  industry. 

OpenFlight ’s  main  strength  is  its  organized  hierarchy  of  polygonal  objects,  with  generous 
support  for  textures  and  material  properties.  Since  all  simulation  environments  are  typically 
represented  as  a  collection  of  polygons,  this  approach  provides  the  most  flexibility.  OpenFlight 
also  provides  several  efficiency  features,  such  as  global  vertex  lists,  normal  lists,  and  color 
tables,  in  addition  to  being  a  binary  format  to  conserve  space.  OpenFlight  also  includes  features 
to  improve  display  speed,  such  as  level-of-detail  information  and  bounding  boxes.  OpenFlight  is 
missing  capabilities  such  as  linestyles  and  external  links  to  text  and  pictures. 

POV-Rav  rPOVi.  51.7% 

The  POV  file  format  was  developed  for  use  in  Persistence  Of  Vision’s  Ray-Tracer  (POV- 
Ray),  a  software  program  developed  by  the  POV  Team  (an  independent  group  of  software 
developers)  that  generates  high-fidelity  images  from  3D  models.  As  with  most  ray-tracers, 
POV-Ray  does  not  display  3D  models  in  real-time,  but  instead  is  used  to  generate  a  single  image 
of  a  3D  model  with  extreme  detail,  or  a  series  of  detailed  images  that  can  be  replayed  like  a 
movie.  POV-Ray  primarily  uses  mathematical  objects  like  spheres  and  cones  rather  than 
arbitrary  groups  of  polygons  because  mathematical  objects  are  easier  to  process  in  the  ray¬ 
tracing  algorithms. 

The  POV  file  format  supports  enough  features  to  be  useful  to  the  aircraft  maintenance 
task,  with  support  for  polygons,  object  groupings,  text,  and  many  types  of  3D  objects.  It  also 
supports  textures,  material  properties,  and  arbitrary  colors.  However,  the  real  strengths  of  POV 
lie  in  its  extensive  support  of  surface  properties  like  transparency  and  reflectivity,  and  extensive 
camera  properties  like  focal  blur  and  atmospheric  attenuation,  which  were  deemed  unnecessary 
for  the  aircraft  maintenance  task.  However,  like  OpenFlight,  the  POV  file  format  does  not  have 
the  capability  for  linestyles  or  external  links  to  text  and  pictures. 
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Initial  Graphics  Exchange  Specification  riGES).  50.5% 


The  IGES  file  format  is  an  ANSI  standard  maintained  by  the  IGES/PDES  organization. 
The  function  of  IGES  is  to  set  up  a  series  of  structures  to  exchange  product  definition  data.  In 
other  words,  it’s  a  file  format  mainly  used  by  computer-aided  design  and  manufacturing  systems 
(CAD/CAM). 

The  IGES  format  covers  most  of  the  criteria  basics,  such  as  polygons,  groupings,  lines 
and  linestyles.  Due  to  its  CAD/CAM  background,  it  satisfies  most  of  the  basic  criteria,  covers  an 
incredible  number  of  mathematical  objects,  and  has  a  strong  measurement  unit  criteria  score. 
Unfortunately  (also  due  to  its  background),  it’s  weak  in  many  of  the  criteria  needed  to  produce  a 
realistic  simulated  environment,  such  as  material  properties  and  textures. 

3D  Studio  GDSk  43.2% 

The  3DS  file  format  is  a  proprietary  format  owned  by  AutoDesk.  Made  popular  by  the 
modeling  software  3D  Studio  (and,  to  a  lesser  extent,  AutoCAD),  3DS  is  a  fairly  commonplace 
file  format.  Although  files  using  the  (binary)  format  are  commonplace,  it’s  surprisingly  difficult 
to  obtain  the  file  format  specifications  from  AutoDesk.  Once  obtained,  3DS  shows  its  physical 
layout  of  the  file  format  as  a  series  of  “chunks”  which  can  have  a  hierarchical  structure. 

While  3DS  is  popular,  it  doesn’t  offer  much  to  this  investigation.  It  has  limited  polygon 
and  grouping  capabilities,  and  has  the  full  criteria  for  links  to  pictures,  color  information, 
transparency,  and  viewing  information.  It  cannot  handle  lines,  text,  external  links  to  text,  or 
mathematical  objects. 

Drawing  Interchange  Format  (DXF).  41.6% 

Similar  to  IGES,  DXF  is  used  for  CAD  applications;  specifically,  DXF  is  the  internal 
structure  for  AutoDesk’s  popular  AutoCAD  software.  Previously  only  a  2D  format,  DXF  first 
had  its  capabilities  extended  in  1985  to  encompass  simple  3D  entities.  Further  revisions  of  the 
format  have  extended  the  3D  DXF  so  that  it  can  contain  more  complex  entities.  The  physical 
layout  of  the  file  format  is  well-defined,  with  certain  sections  including  AutoCAD-specific 
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information,  and  other  sections  containing  information  on  the  2D/3D  structures  to  be  displayed 
in  the  simulated  environment. 

Due  to  its  CAD  background,  DXF  provides  a  strong  basis  for  CAD-related  criteria  such 
as  lines,  text,  measurement  units,  and  points.  However,  like  IGES,  it  lacks  many  of  the  criteria 
needed  to  create  a  convincing  simulated  environment,  such  as  color  information,  material 
properties,  normals,  animations,  and  light  sources. 

Wavefront  tOBJi.  40.8% 

The  OBJ  file  format  was  developed  by  Wavefront.  OBJ  is  a  very  stable  format,  and  can 
be  as  stored  as  ASCII  format  files  (.obj)  and  binary  format  files  (.mod).  OBJ  files  define 
geometric  and  other  properties  for  objects  in  Wavefront’s  Advanced  Visualizer  application.  The 
file  format  has  a  free-form  organization.  OBJ  files  store  geometric  objects  composed  of  lines, 
polygons,  and  free-form  curves  and  surfaces.  As  far  as  support  for  its  3D  file  format  is 
concerned.  Wavefront  maintains  a  toll-free  support  number  and  Bulletin  Board  System  (BBS) 
for  its  customers.  There  are  also  many  Wavefront  user  groups. 

Although  the  OBJ  file  format  is  used  by  many  popular  applications,  as  a  3D  file  format  it 
has  many  shortcomings.  Among  them  are  lack  of  text  support;  provisions  for  specular,  diffuse, 
and  ambient  material  properties  only;  lack  of  animation  or  articulation;  and  no  support  for 
wireframe  or  light  sources. 

Neutral  File  Format  /  Extended  Neutral  File  Format  (NFF  /  ENFF),  27.2% 

The  Neutral  File  Format  is  maintained  by  Eric  Haines  of  3D/Eye  Inc.  The  format  was 
written  to  have  minimal  constructs  that  could  be  used  for  testing  rendering  algorithms.  It  is 
intended  to  be  used  with  the  Standard  Procedural  Database  (SPD)  software  package.  The  first 
draft  of  the  format  was  created  in  October  1988,  and  revisions  and  extensions  were  made  until 
the  most  recent  revision  in  April  1993.  Because  the  format  is  minimal,  its  ASCII  text  files  are 
simple  to  create  and  read. 

The  format  supports  most  of  the  basic  constructs  necessary  to  create  geometry,  colors, 
shading,  lighting,  and  a  viewpoint.  The  format  is  missing  many  higher-level  constructs  such  as 
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groupings,  lines,  text,  textures,  measurement  units,  points,  switches,  levels  of  detail  (LOD), 
animations  and  articulations.  NFF  is  very  useful  for  writing  simple  test  cases  for  testing 
rendering  algorithms,  but  is  missing  many  of  the  constructs  necessary  for  use  as  a  general- 
purpose  3D  graphics  file  format. 


47 


3D  VIEWERS 


Identification  of  3D  Viewers 

The  purpose  of  this  phase  of  the  3D  Graphics  Investigation  was  to  find  the  ideal  3D 
viewer:  a  package  that  would  allow  the  data  contained  in  a  3D  file  format  to  be  presented 
onscreen  in  a  3D  simulated  environment.  Such  a  package  can  be  found  in  nearly  any  field  that 
uses  computers  and  data  visualization. 

The  following  types  of  software  packages  were  considered  as  likely  candidates  for  an 
ideal  3D  viewer:  3D  modeling  software,  CAD  packages  (including  design  prototyping  systems 
as  well  as  drafting  and  architectural  systems),  and  virtual  reality  (most  commonly  VRML) 
software.  Upon  initial  inspection,  these  fields  appeared  to  offer  products  that  closely  matched  the 
needs  of  the  investigation.  However,  even  when  limited  to  these  areas,  the  marketplace  offered 
several  hundred  software  packages  from  which  to  choose.  To  make  an  objective  choice  between 
the  good  and  mediocre  3D  viewers,  the  necessary  criteria  for  a  3D  viewer  were  defined.  The 
criteria  were  based  on  three  fundamental  themes: 

1)  The  ideal  viewer  will  use  graphics  technology  that  produces  a  realistic,  simulated  3D 
environment  that  could  be  manipulated. 

2)  The  ideal  viewer  will  contain  all  the  features  requested  by  the  group  of  maintainers 
during  the  CTA. 

3)  The  ideal  viewer  will  contain  all  the  features  requested  by  HESR  representatives, 
including  low  cost  and  the  ability  to  interface  with  common  WWW  browsers.  (It  was 
assumed  that  future  electronic  TOs  would  be  authored  in  HTML,  allowing  both  the 
3D  graphics  and  textual  TO  information  to  be  tightly  integrated.  In  addition,  this 
software  would  reside  on  a  large  number  of  machines,  indicating  the  need  for  a  low 
price.) 

The  target  hardware  platform  for  the  3D  viewer  was  an  Intel  Pentium-based  laptop 
computer  operating  at  200  Mhz  on  a  Windows  platform. 

3D  Viewers  Identified 

Using  the  WWW  as  the  primary  source  of  information,  the  team  identified  19  candidate 
3D  viewers,  listed  in  Table  5.  The  first  colunrn  of  the  table  lists  the  name  and  version  of  the 
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viewer,  the  second  column  lists  the  name  of  the  company  responsible  for  developing/supporting 
the  viewer,  the  third  column  indicates  whether  the  viewer  is  a  plugin,  and  the  last  column  lists  a 
universal  resource  locator  (URL)  link  for  product  information  and  demonstration  versions.  Free 
evaluation  copies  were  available  for  all  but  one  of  the  viewers  (3D  Studio  Max.) 

One  popular  CAD  software  package  absent  from  the  list.  AutoCAD  by  Autodesk,  was 
considered  and  researched  but  not  evaluated.  The  team  conducted  a  meeting  with  a  local  value 
added  re-seller  (VAR)  of  the  AutoCAD  product.  After  thoroughly  explaining  the  functionality 
required  of  the  3D  viewer  for  the  task,  the  team  and  the  VAR  collectively  determined  that  the 
AutoCAD  product  was  not  a  viable  candidate  for  this  study.  The  3D  capability  of  AutoCAD  is 
centered  primarily  around  wireframe  models. 

Table  5.  Identified  3D  Viewers. 


Viewer  Name 

Company  Name 

Plug-in 

URL  Link 

3D  View  2.1 

Actify,  Inc. 

No 

http://www.actify.com/ 

Ouicktime  VR  2.1.2 

Apple  Inc. 

Yes 

http://www.apple.com/quicktime/qtvr/ 

Cosmo  Player  2.0 

Silicon  Graphics 

Yes 

http://cosmosoftware.com/ 

Community  Place 

VRML  2.0  D2 

Sony 

Yes 

http://vs.sony.co.jp/ 

V-Realm  Browser  1.1  Build  1 

Integrated  Data 
Systems 

Yes 

http://www.ligos.com/ 

World  View  2.0 

Intervista  Software, 
Inc. 

Yes 

http  ://w  w  w .  intervista.com/worldview 

Platinum  WIRL  1 .0 

Beta  G 

Platinum 

Techonology,  Inc. 

Yes 

http://www.platinum.com/products/appdev/vrea 
m/wirl  ps.htm 

NetAnimator  1.0 

Release  3 

Geometric  Software 
Services  Co. 

Yes 

http://www.gsslco.com/ 

SDSpace  Assistant  2.0 

Template 

Graphics  Software 

Inc. 

No 

http://www.tgs.com/ 

Beyond  3D 

Extreme  2.01  beta 

Uppercut  Software 

No 

http://www.beyond-3d.com/ 

OZ  Virtual  2.0b3 

OZ  Interactive 

Yes 

http://www.oz.com/ 

Realiview  / 

Realimation  4.3 

Datapath 

Yes 

http://www.realimation.com/ 

ConceptCAD  3.1  TestDrive 

Virtus  Corporation 

Yes 

http  ://w w  w .  virtus  .com/ 

Virtus  Player  3.0 

Virtus  Corporation 

No 

http://www.virtus.com/ 

Cult3D  1.1 

Cycore  Computers 

Yes 

http  ://cuIt3  d .  com/ 

Community  Client  3D  3.02 

Blaxxun  Interactive 

Yes 

http  ://w  ww.blaxxun.com/ 

ActiveCGM  Browser  5.0 

InterCAP  Graphics 
Systems 

Yes 

http://www.intercap.com/ 

3D  Studio  Max  1.2 

Kinetix  /  Autodesk 

No 

http://ktx.com/ 

World  Up  Player 

Release  4 

Senses 

Yes 

http://www.sense8.com/products/download.htm 

1 

Quick3D  Beta 

PlasticThought 

Yes 

http://www.PlasticThought.com/Quick3D_abou 

t.html/ 
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3D  Viewer  Evaluation  Criteria 


As  was  done  for  the  file  format  evaluations,  a  set  of  weighted  evaluation  criteria  was 
developed  to  evaluate  the  3D  viewers.  Table  6  lists  the  evaluation  criteria  used.  The  same  format 
and  process  used  to  develop  the  evaluation  criteria  for  the  file  formats  was  used  to  develop  the 
viewer  evaluation  criteria. 

Similar  to  the  evaluation  of  the  file  formats,  a  set  of  weighted  criteria  was  developed. 
This  list  of  3D  viewer  criteria  was  created  by  focusing  on  the  following  needs: 

1)  The  3D  viewer  has  some  feature  which  enhances  realism  of  the  simulated 
environment. 

2)  The  3D  viewer  has  some  feature  which  allows  for  manipulation  of  the  simulated 
environment. 

3)  The  3D  viewer  has  some  feature  which  enhances  its  use,  its  distribution,  or  its  ability 

to  be  modified. 

As  was  completed  with  the  file  formats,  each  of  the  criteria  was  given  a  weight  factor 
between  1  and  10,  depending  on  its  relative  importance.  These  weights  were  determined  by 
features  requested  by  HESR  representatives,  requests  from  the  group  of  maintainers  during  the 
CTA,  and  the  need  to  produce  a  realistic,  modifiable  simulated  environment  that  could  be 
manipulated.  Next,  each  of  the  criteria  was  further  broken  down  into  one  or  more  sub-criteria. 
For  a  detailed  explanation  of  how  the  following  table  was  used  to  score  each  of  the  viewers 
please  refer  to  the  section  titled,  “Identification  of  3D  File  Formats.” 
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Table  6.  3D  Viewer  Evaluation  Criteria. 

Bolded  criteria  indicate  “first  pass”  scoring  system;  all  criteria  were  used  in  “final  pass” 

scoring  system 


Criteria 

wt 

. . . 

Cost 

10 

□  (5)  Free:  The  3D  viewer  is  free. 

□  (4)  Low  price:  The  3D  viewer  retails  for  less  than  $100. 

□  {} )  Hi2h  vrice:  The  3D  viewer  retails  for  less  than  $500. 

Perspective  View 

10 

□  (2)  Perspective  view:  The  viewer  allows  the  simulated  environment  to  be 
shown  in  a  perspective  view. 

□  (2)  Clip  planes:  The  far  and  near  clipping  planes  can  be  modified. 

□  (2 )  Single  view  manipulation:  A  single  movement  paradigm  is  supported. 

□  (2)  Multiple  view  manipulation:  Multiple  movement  paradigms  are 
supported. 

□  FOV:  The  field  of  view  can  be  modified. 

□  (})  Collision  detection:  The  viewer  supports  the  ability  to  both  move 
though  and  be  blocked  by  primitives. 

Stability/ 

Robustness 

10 

□  (lO/X)  Test  number  X:  The  viewer  never  quits  unexpectedly  or  crashes 
while  running  a  test. 

Extensibility 

10 

□  (5)  User  Extensibility:  The  user  can  add  in  new,  user-created  functionality 
to  the  capabilities  of  the  viewer. 

□  (5)  Vendor  Extensibility:  The  user  can  add  in  new,  vendor-created 
functionality  to  the  capabilities  of  the  viewer. 

Groupings 

10 

□  (3 )  Groupings:  In  the  viewer,  one  or  more  primitives  can  be  considered  as  a 
single  group. 

□  (2)  Named  Groupings:  The  descriptive  group  names  can  be  displayed  to  the 
user. 

□  (2)  Manipulation:  The  user  can  place  the  groupings  at  other  locations  in  the 
world. 

□  (2)  Show  groupings:  Groupings  can  be  displayed  or  hidden  from  view. 

□  (I }  Articulation:  Groupings  can  have  articulation  limits. 

Polygon  Support 

9 

□  (8)  Polygon  support:  The  viewer  supports  displaying  the  environment  as  a 
set  of  3D  polygonal  surfaces. 

□  (2 )  Backfacing:  The  viewer  supports  backfacing. 

Run  Time  Speed 

8 

□  (lO/X)  Test  number  X:  The  viewer  displays  the  environment  within  a  set  of 
update  rate  limits.  Update  rates  1  Hz  or  higher  are  considered  acceptable. 

WWW  Plug-In 

8 

□  (8)  Plug-In:  The  viewer  is  a  plug-in  to  a  World  Wide  Web  browser. 

Q  (2)  Stand-alone:  The  viewer  can,  in  addition  to  running  as  a  plug-in,  run  as 
a  stand-alone  viewer. 

Orthogonal  View 

7 

□  (3)  Orthogonal  view:  The  viewer  allows  the  simulated  environment  to  be 
shown  in  an  orthogonal  view. 

□  (2)  Camera  location:  The  viewer  allows  the  camera  location  to  be  moved 
in  an  orthogonal  view. 

□  (2)  Camera  rotation:  The  viewer  allows  the  camera  rotation  to  be  changed 
in  an  orthogonal  view. 

□  (2)  Size  of  view:  The  viewer  allows  the  capability  to  zoom  in  and  out  from 
the  position  of  an  orthogonal  view. 

□  (I)  Clip  planes:  The  far  and  near  clipping  planes  can  be  modified  in  an 
orthogonal  view. 
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Criteria 

Wt. 

Sub  Criteria 

Software  Support 

7 

□  (4)  Support:  The  group  that  created  the  viewer  offers  support  on  viewer 
capabilities. 

□  (3)  Unlimited  free  support:  The  group  that  created  the  viewer  offers  free 
support  and  instructions  on  viewer  capabilities 

□  (2)  User  support:  A  subset  of  users  offers  freely  available  support  and 
instructions  on  viewer  capabilities. 

□  (I)  Limited  free  support:  The  group  that  created  the  viewer  offers  limited 
free  support  and  instructions  on  viewer  capabilities. 

Picture  Support 

7 

□  (10)  Picture  support:  The  viewer  supports  the  ability  to  spawn  an 

additional  window  which  contains  2D  bitmap  information.  The  picture  file 
information  is  associated  with  an  object  or  an  action  in  the  environment. 

Text  Support 

7 

□  (10)  Text  support:  The  viewer  supports  the  ability  to  spawn  an  additional 

window  which  contains  text  information.  The  text  information  is 
associated  with  an  object  or  an  action  in  the  environment. 

Text  Support  in 
Environment 

7 

□  (4)  Text:  The  viewer  can  display  text  in  the  simulated  environment. 

□  (2)  Size:  The  text’s  size  can  be  displayed  in  the  viewer. 

□  Font:  The  text’s  font  can  be  displayed  in  the  viewer. 

□  Justification:  The  text’s  justification  can  be  displayed  in  the  viewer. 

□  Orientation:  The  text’s  orientation  can  be  displayed  in  the  viewer. 

□  (1 )  Color:  The  text’s  color  can  be  displayed  in  the  viewer. 

Line  Support 

7 

□  (7)  Line:  The  viewer  can  display  the  lines  in  the  environment. 

□  (2)  Pattern:  The  viewer  can  display  patterns  on  lines. 

□  (1 )  Thickness:  The  viewer  can  display  the  thickness  of  lines. 

Undo  Feature 

7 

□  (6)  Multiple:  The  viewer  can  step  back  though  more  than  one  action 

□  (4  ]  Sinsle:  The  viewer  can  step  back  a  single  action. 

Load  Time  Speed 

7 

□  (lO/X)  Test  number  X:  The  viewer  can  set  up  and  load  in  the  test  case 

environments.  A  set  of  limits  that  represent  the  times  that  the  viewer  can  do 
these  tasks  will  be  created.  Load  time  of  less  than  1  minute  is  acceptable. 

Transparency 

Support 

6 

□  (7)  Transparency:  The  viewer  allows  primitives  to  be  partially  or  totally 
transparent  when  drawn  in  the  environment. 

□  (2)  Adjust:  Transparency  levels  of  primitives  can  be  adjusted  in  the  viewer. 

Q  (1)  Turn  on/off:  Transparency  levels  of  the  primitives  can  be  turned  on  or 

off  If  the  adjust  feature  is  marked,  this  feature  is  automatically  marked. 

Shading  Support 

6 

Q  (4)  Interpolation:  The  viewer  allows  primitives  to  be  shaded  by  normal  or 
intensity  interpolation  methods. 

□  (3)  Radiosity:  The  viewer  allows  primitives  to  be  shaded  by  radiosity 
methods. 

□  (2)  Flat:  The  viewer  allows  primitives  to  be  shaded  by  flat  shading 
methods. 

User  Friendly 

6 

□  (10)  User  Friendly:  The  viewer  is  ranked  as  either  ‘friendly’  or  ‘not 

friendly’  by  a  user  who  is  not  familiar  with  the  viewer.  The  user  will  be  run 
through  a  set  of  test  cases. 

Multiple 

Simultaneous  Views 

6 

□  (4)  Two  or  more  views:  Two  or  more  views  of  the  environment  can  be 
shown  simultaneously  by  the  viewer. 

□  (3)  Three  or  more  views:  Three  or  more  views  of  the  environment  can  be 
shown  simultaneously  by  the  viewer. 

□  (2)  Four  or  more  views:  Four  or  more  views  of  the  environment  can  be 
shown  simultaneously  by  the  viewer. 

□  (] )  Five  or  more  views:  Five  or  more  views  of  the  environment  can  be 
shown  simultaneously  by  the  viewer. 
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'Criteria 

Wt. 

Color  Support 

6 

□  {6)  16-bit  or  greater  color.  The  viewer  can  show  a  view  of  the  environment 
in  16-bit  (65536  colors)  or  a  larger  number  of  colors. 

□  (3)  8-bit  color:  The  view  can  show  a  view  of  the  environment  in  8-bit  color 
(256  colors). 

□  (1)  4-bit  color:  The  viewer  can  show  a  view  of  the  environment  in  4-bit 
color  (16  colors). 

Mathematical 
Primitive  Support 

6 

□  (2 }  Common  3D  objects:  The  viewer  can  display  spheres  or  boxes. 

□  (2)  Less  common  3D  objects:  The  viewer  can  display  torii,  or  discs,  or 
cylinders,  or  cones,  or  surfaces  of  revolution,  or  extruded  surfaces. 

□  (2)  Common  2D  objects:  The  viewer  can  display  ellipses  or  n-sided  figures. 

□  Less  common  2D  objects:  The  viewer  can  display  splines  or  arcs. 

□  Rare  3D  objects:  The  viewer  can  display  fractals  or  mathematical  surfaces. 

□  (1)  Constructive  Solid  Geometry:  The  viewer  can  display  objects  created 
out  of  logical  operations  (such  as  AND,  OR)  on  a  set  of  mathematical 
objects. 

Measurement  Units 

6 

□  (4)  Options:  The  viewer  can  convert  a  primitive’s  units  of  measure  from 
one  system  to  another. 

□  (2)  Units  of  Measure:  The  view  can  display  a  primitive’s  units  of  measure. 

□  (2)  Dimensions:  The  viewer  can  display  a  primitive’s  units  dimensions. 

□  (2)  Turn  on/off:  The  viewer  allows  the  user  to  turn  the  display  of 
measurement  units  on  and  off. 

Texture  Support 

5 

□ .  (6)  2D  Textures:  The  viewer  can  support  the  display  of  two-dimensional 
textures  on  the  surface  of  a  primitive. 

□  (3)  Draw  style:  The  viewer  can  support  the  display  of  a  particular  repetitive 
pattern  on  the  surface  of  a  primitive.  If  a  viewer  has  2D  textures,  it  has  this 
sub-criterion. 

□  (1)  3D  Textures:  The  viewer  can  support  the  display  of  three-dimensional 
textures  on  the  surface  primitive. 

Animation 

5 

□  (4)  3D  Transformation:  The  viewer  can  support  the  display  of  an  animation 
sequence,  which  is  created  by  changing  the  transformation  parameters  of 
primitives. 

□  (4)  3D  Switches:  The  viewer  can  support  an  animation  display 

□  2D  Animation:  The  viewer  can  support  the  display  of  an  animation 
sequence,  which  is  created  by  sequencing  through  a  series  of  texture  maps 
or  material  property  patterns. 

□  (1)  3D  Morphing:  The  viewer  can  support  the  display  of  an  animation 
sequence,  which  is  created  by  interpolating  between  different  versions  of  a 
primitive’s  geometry. 

Error  Checking 

5 

□  (5)  Corrupt  files:  The  viewer  will  give  a  meaningful  error  or  warning 
message  when  reading  in  corrupt  files. 

□  (5)  File  versions:  The  viewer  will  give  a  meaningful  error  or  warning 
message  when  reading  in  files  of  a  different  version  than  the  viewer 
supports. 

Multiple  File  Load 

4 

□  (70)  Multiple  file  load:  The  viewer  can  load  two  or  more  un-nested, 

independent  files  into  the  same  environment. 

Slices/  Clip  Planes 

4 

□  (5)  Clip  planes:  Additional  clipping  planes  (other  that  the  hither  and  yon 
planes)  are  supported  in  the  viewer. 

□  (4)  Options:  The  position  of  the  clip  plane  can  be  modified  in  the  viewer. 

□  (1 )  Turn  on/off:  The  clip  planes  can  be  turned  on  or  off  in  the  viewer. 

Render  Order 

4 

□  (5)  File  format  implementation  :  The  3D  viewer  can  render  a  set  of 
primitives  in  the  order  specified  by  the  file  format. 

□  (5)  User  implementation:  The  3D  viewer  can  render  a  set  of  primitives  in 
the  order  specified  by  the  user. 
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Criteria  , 

iiaii 

'Sub'Criteria' '  . .  . 

Wireframe  Support 

3 

□  (5)  Wireframe:  The  viewer  supports  the  displaying  of  the  environment  as  a 
set  of  wireframe  surfaces. 

Q  (2)  Depth  cueing:  The  wireframe  lines  are  shaded  in  some  way  to  indicate 
depth, 

□  (2)  Pattern:  The  viewer  allows  adjustment  of  the  patterns  on  wireframe 
lines. 

□  (1 )  Thickness:  The  viewer  allows  adjustment  of  the  thickness  of  wireframe 
lines. 

Documentation 

3 

□  (4)  Electronic  copy:  Documentation  with  a  computer  search  capability  is 
available. 

□  (4)  Tutorials:  A  tutorial  of  how  to  use  the  viewer  is  available. 

□  (2)  Hard  Copy:  Documentation  in  printed  format  is  available,  or  an  option 
to  print  the  electronic  copy  is  available. 

Cross-Platform 

2 

□  (10)  Cross-Platform:  Viewer  is  available  on  multiple  major  platforms 

(UNIX,  Windows,  MacOS). 

Ease  of  Installation 
and  Upgrade 

2 

□  (5)  Install  program:  Some  type  of  automatic  installation  program  handles 
the  3D  viewer  installation. 

□  (3)  All  components:  The  viewer  does  not  require  additional  separate 
software  packages  or  libraries  to  work. 

□  (2 )  Instruction  set:  A  detailed,  step  by  step  set  of  instructions  exists  for 
installing  the  3D  viewer. 

Point  Support 

2 

□  (6)  Display:  The  viewer  can  display  points  in  the  environment. 

□  (4)  Size:  The  point  sizes  can  be  displayed  in  the  environment. 

Sound  Support 

2 

□  (10)  Sound  Support:  The  viewer  can  play  digitized  sound  information.  The 

sound  file  information  is  associated  with  an  object  or  an  action  in  the 
environment. 

Light  Sources 

2 

□  (4)  Infinite  light  source:  The  3D  viewer  can  display  infinite  light  sources. 

□  (4)  Local  light  source:  The  3D  viewer  can  display  local  light  sources. 

□  Spotlight  light  source:  The  3D  viewer  can  display  spotlight  light  sources. 

□  (1)  Material  Properties:  The  3D  viewer  can  support  the  display  of  all  the 
material  properties  in  the  file  format  “Material  Properties”  category. 

File  Compression 
Support 

1 

□  (10)  Compressed  flies:  The  viewer  can  read  in  3D  data  files,  which  have 

had  their  data  compressed. 

_ ^JL _ _ _ 

Efficiency  Issues 
with  Large  3D 
Models  i 

1 

□  (10)  Efficiency  Issues:  The  viewer  implements  some  sort  of  efficiency 

scheme  for  handling  large  3D  models,  such  as  LOD,  or  bounding  or 
culling. 

3D  Viewers  Evaluation  Results 


A  two-pass  process  was  used  to  evaluate  the  identified  3D  viewers.  During  the  first  pass, 
each  of  the  viewers  identified  in  Table  5  was  evaluated  against  a  subset  of  the  evaluation  criteria 
found  in  Table  6.  The  evaluation  criterion  used  during  the  first-pass  evaluation  are  identified  by 
bold  type  in  Table  6.  These  “first-pass”  criteria  were  chosen  by  their  relatively  high  weight  as 
well  as  the  short  amount  of  time  it  took  to  determine  their  answer.  During  the  second  pass,  all  of 
the  evaluation  criteria  from  Table  6  were  used  to  score  the  viewer. 


The  primary  reason  for  the  first-pass  evaluation  was  to  determine  which  of  the  19  viewers 
should  be  fully  evaluated.  Due  to  the  duration  and  dollar  value  of  the  effort,  all  of  the  19  viewers 
could  not  be  fully  evaluated.  The  first-pass  evaluation  process  took,  on  average,  a  hour  to 
complete.  The  second-pass  evaluation  process  for  each  viewer  took  an  average  of  six  hours  - 
although  the  time  was  reduced  to  two  hours  for  VRML  viewers  once  a  test  bed  of  VRML  files 
was  established.  Table  7  summarizes  the  results  of  the  first-pass  evaluation.  The  first  column  of 
the  table  identifies  the  evaluated  viewer  and  the  second  column  contains  the  normalized  (0-100) 
score. 


Table  7.  First  Pass  Evaluation  Results 


Viewer 

Normalized 

Score 

Cosmo  Player  v2.1  Beta 

71 

3D  Studio  Max  1 .2 

70.3 

Realiview  /  Realimation  4.3 

64.7 

Community  Client  3D  3.02 

64.0 

Platinum  WIRL  1 .0  Beta  G 

60.3 

World  View  2.0 

58.5 

Community  Place  VRML  2.0  D2 

54.4 

V-Realm  Browser  1.1  Build  1 

54.1 

ActiveCGM  Browser  5.0 

48.5 

ConceptCAD  3,1  TestDrive 

46.5 

Quicktime  VR  2,1.2 

40.8 

OZ  Virtual  2.0b3  H 

39.2 

World  Up  Player  Release  4 

38.5 

3D  View  2.1 

34.0 

3Dspace  Assistant  2.0  Template 

33.1 

Virtus  Player  3.0 

30.4 

NetAnimator  1 .0  Release  3 

27.5 

Beyond  3D  Extreme  2.01  beta 

26.3 

CultSD  1.1 

26.0 

Ouick3D  Beta 

19.2 

Based  on  the  results  of  the  first-pass  evaluation  results,  six  viewers  were  selected  for  the 
full  review.  The  four  highest  ranking  viewers  (Cosmo  Player,  3D  Studio  Max,  Realiview  / 
Realimation,  Community  Client  3D)  were  chosen  to  find  the  best  product  to  use  in  the  study. 
Two  medium-to-lower  ranking  viewers  (V-Realm  Browser,  OZ  Virtual)  were  chosen  to  validate 
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whether  or  not  conducting  a  first-pass  test  would  indicate  (roughly)  the  score  of  a  final  test.  As 
shown  from  the  scores  in  the  first-pass  and  final  evaluation  scores,  viewers  that  rank  highly  in 
the  preliminary  scoring  system  tend  to  score  highly  in  the  final  scoring  system;  low  preliminary 
scorers  tend  to  score  low  in  the  final  scoring  system. 

Table  8  summarizes  how  the  six  selected  viewers  performed  against  the  full  set  of 
evaluation  criteria.  The  six  viewers  are  listed  across  the  top  of  the  table  starting  in  the  second 
column.  The  first  column  identifies  the  criterion.  The  raw  score  for  each  of  the  criteria  is 
provided  starting  in  the  second  column  and  second  row,  where  the  raw  score  is  defined  to  be  the 
sum  of  the  satisfied  sub-criteria  times  the  weight  for  the  particular  criterion.  The  last  two  rows  of 
Table  8  list  the  raw  score  total  and  the  normalized  score  for  each  of  the  file  formats.  The  raw 
score  total  was  calculated  by  adding  all  of  the  raw  scores  for  each  of  the  criterion.  Dividing  the 
raw  score  by  the  total  possible  points,  in  this  case  2180,  produced  a  normalized  score  for  each  of 
the  file  formats  (a  score  between  0-100). 
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Table  8.  3D  Viewer  Evaluation  Results 


SDStuidio 

MaxTl.2 

Gosmo  Pl^er  : 
v2.1  Beta 

Community 
Client  34)  3^0 

Realiview/ 
Kealimation  43 

OZ  Virtual 
2.0b3 

V-Realm 
Browser  14 

Cost 

0 

100 

100 

100 

100 

100 

Perspective  View 

70 

70 

70 

100 

70 

Stability/  Robustness 

100 

100 

100 

100 

80 

0 

Extensibility 

100 

50 

50 

0 

0 

0 

Groupings 

100 

60 

60 

50 

20 

30 

Polveon  Support 

90 

90 

90 

90 

90 

90 

80 

80 

80 

80 

64 

0 

0 

64 

64 

80 

80 

80 

63 

0 

70 

0 

Software  Support 

70 

56 

49 

56 

■OH 

Picture  Support 

0 

70 

70 

70 

0 

70 

Text  SuDDort 

70 

70 

70 

0 

0 

EBnsRiniSRir^^Hi 

63 

56 

0 

63 

Line  Support 

49 

49 

0 

0 

0 

Undo  Feature 

70 

0 

0 

0 

0 

■iiiiii  IIIIII1 

70 

70 

70 

70 

56 

0 

Transparency  Support 

60 

42 

42 

42 

42 

0 

Shading  Support 

42 

24 

42 

36 

42 

42 

User  Friendly 

60 

60 

60 

60 

60 

0 

Multiple  Simultaneous 
Views 

54 

0 

0 

0 

0 

Color  Support 

60 

60 

24 

60 

60 

60 

Mathematical  Primitive 
Support 

54 

30 

30 

30 

30 

30 

Measurement  Units 

36 

0 

0 

0 

0 

0 

45 

45 

45  1 

45 

45 

45 

Animation 

50 

45 

20 

0 

Error  Checking 

0 

■■SHi 

0 

50 

25 

Multiple  File  Load 

40 

0 

0 

0 

0 

0 

Slices/  Clip  Planes 

0 

0 

0 

0 

0 

0 

Render  Order 

0 

0 

0 

0 

0 

0 

Wireframe  Support 

21 

0 

15 

15 

0 

15 

Documentation 

30 

■■KSlHi 

18 

18 

0 

Cross-Platform 

0 

0 

\mmmm 

0 

Ease  of  Installation  and 
Upgrade 

20 

20 

20 

20 

20 

20 

Point  Support 

12 

12 

12 

0 

0 

0 

Sound  Support 

0 

20 

20 

0 

20 

0 

Light  Sources 

18 

20 

20 

20 

18 

0 

File  Compression 
Support 

0 

10 

10 

0 

0 

10 

Efficiency  Issues  with 
Large  3D  mddels 

10 

10 

10 

10 

10 

10 

:  Raw  Score  Total  ^  ^ 

1607 

;  ;:1586 

1405  • 

1361 

1013  : 

760-  f . 

iiM 

^  12 A 

:  64.4 

62.4 

■-;i46.5i,S" 

■a  .34;9v  .:  . 
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Descriptions  of  3D  Viewers 


3D  Studio  MAX  1.2.  73.7% 

3D  Studio  MAX  is  a  software  package  developed  and  sold  by  Kinetix,  a  division  of 
Autodesk  Inc.  This  package  has  roots  in  the  computer-aided  design  market,  and  retains  many  of 
the  features  of  CAD  software,  but  has  been  expanded  to  accommodate  the  needs  of  computer 
animators.  3D  Studio  MAX  is  geared  to  the  expert  user,  and  its  user-interface  can  be  rather 
daunting  to  a  novice,  but  it  has  many  features  useful  for  an  aircraft  maintenance  task,  and  can  be 
extended  via  third-party  plug-ins. 

The  main  strengths  of  3D  Studio  MAX  are  its  support  for  polygons,  named  object 
groupings,  multiple  independent  views,  orthogonal  views,  measurement  units,  numerous 
mathematical  object  types,  and  its  extensibility.  It  faltered  somewhat  on  links  to  external  text, 
since  its  support  was  limited  to  a  single  window  of  unformatted  text  that  had  to  be  input  via  3D 
Studio  MAX,  rather  than  linking  to  an  existing  file.  Other  drawbacks  included  a  high  price  tag, 
an  inability  to  associate  a  2D  picture  with  an  object  or  action,  and  an  inability  to  be  run  as  a  plug¬ 
in  a  WWW  browser. 

Cosmo  Player  2.1. 72.8% 

Cosmo  Player  2.1  is  a  free,  high-performance,  cross-platform  viewer  for  applications 
written  in  VRML  2.0,  the  open  standard  for  3D  content  on  the  Web.  Cosmo  Player  was 
developed  by  Silicon  Graphics,  Inc.  (SGI)  to  be  fully  compliant  with  VRML  2.0  specifications, 
plus  to  take  full  advantage  of  popular  user-side  graphics  “turbocharging  such  as  Direct3D, 
OpenGL,  and  AGP  technology.  The  viewer  interface  is  easy  on  beginners  yet  full  featured  for 
experts. 

With  its  user  friendliness  and  powerful  features,  the  viewer  has  become  very  popular, 
with  millions  of  free  versions  distributed  to  date.  For  users  with  PC  compatibles,  the  viewer  is  a 
plug-in  for  Communicator  4.x,  and  ActiveX  control  for  Internet  Explorer  4.X. 
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Although  Cosmo  Player  is  a  widely-used  VRML  viewer,  it  has  a  few  shortcomings.  It  is 
not  capable  of  displaying  objects  in  an  orthogonal  view,  supports  only  one  view  of  a  scene,  does 
not  support  measurement  units,  and  does  not  support  displaying  objects  in  a  wireframe  view. 

Blaxxun  Community  Client  3D.  64.4% 

Community  Client  3D  (CC3D)  is  a  VRML  viewer  distributed  by  Blaxxun  Interactive. 
Blaxxun  targets  their  Community  Client  family  of  products  to  the  on-line  community.  (Their 
more  advanced  product,  CCpro,  adds  a  number  of  multi-user  environment  features  to  the  features 
available  in  CC3D.) 

CC3D  is  a  plug-in  for  the  Explorer  and  Communicator  Web  browsers.  CC3D  correctly 
reads  in  files  in  VRML  2.0.  It  fully  supports  polygons,  text  and  picture  links  and  almost  all  of 
the  embedded  text  features,  as  well  as  light  sources,  sounds,  compressed  files  and  LOD, 
groupings,  shading,  textures  and  animations.  It  supports  the  basic  features  of  lines,  transparency, 
mathematical  objects,  wireframe,  and  points.  The  CC3D  viewer  does  not  support  an  undo 
feature,  multiple  simultaneous  views,  measurement  units,  error  checking,  multiple  file  loads, 
orthogonal  views,  or  arbitrary  clip  planes. 

Rp.aliview  /  RealiMation  4.3.  62.4% 

Realiview  is  a  free  viewer  distributed  by  RealiMation,  Inc.  RealiMation  is  also  the  name 
given  to  a  suite  of  products  for  visualization,  simulation  and  games.  The  RealiMation  Space 
Time  Editor  (STE)  must  be  used  to  create  the  Realibases  that  can  be  read  and  displayed  by 
Realiview.  The  latest  version  of  the  RealiMation  product  line  is  version  4.3. 

Realiview  is  available  as  both  a  stand-alone  package  or  as  a  plug-in  to  a  Web  browser. 
Realiview  will  run  on  either  a  Windows  95  or  Windows  NT  workstation.  The  interfaces  for  the 
stand-alone  and  the  plug-in,  although  slightly  different,  are  fairly  easy  to  use. 

Realiview  offers  full  support  of  both  perspective  and  orthogonal  viewing.  It  supports 
links  to  external  pictures  and  text  as  well  as  embedded  text  in  the  environment.  It  supports  full 
color  and  lighting  and  basic  shading  and  2D  textures.  For  a  number  of  features,  Realiview 
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provides  basic  support,  but  it  does  not  support  the  more  complex  elements.  Some  of  these 
features  include  groupings,  animations,  mathematical  objects,  and  wireframe. 

Realiview  supports  only  a  single  simultaneous  view,  although  it  is  fairly  easy  to  switch 
between  a  number  of  pre-set  views.  Realiview  does  not  allow  extensibility  by  either  the  user  or 
third-party  vendors,  and  it  does  not  support  an  undo  feature,  measurement  units,  multiple  file 
loads,  arbitrary  clip  planes,  render  order  information,  lines,  or  points. 

OZ  Virtual  2.0  b3.  46.5% 

OZ  Virtual  2.0  was  developed  by  OZ  Interactive.  OZ  Virtual  allows  a  user  to  navigate  a 
3D  world  and  to  create  a  totally  interactive,  multi-user  environment,  which  allows  the  user  to 
communicate  with  people  throughout  the  world.  From  the  startup  3D  world  one  can  navigate  to 
other  worlds  by  double-clicking  hot-spots,  or  open  any  other  3D  world  represented  by  a  VRML 
file. 


The  strengths  of  OZ  Virtual  are  the  ability  to  run  both  stand-alone  and  as  a  plug-in.  It 
will  render  VRML  files,  and  the  viewer  is  free.  However,  OZ  Virtual  has  many  shortcomings.  It 
is  only  partially  VRML  2.0  compliant,  has  limited  support  for  manipulating  groups  of  objects, 
and  has  no  support  for  an  orthogonal  views,  text,  or  line  support.  It  supports  only  one  view  and 
does  not  support  rendering  objects  in  wireframe.  The  viewer  also  is  somewhat  difficult  to  use  to 
navigate  3D  scenes. 

V-Realm  1.1.  34.9% 

The  V-Realm  viewer  is  a  VRML  viewer  produced  by  Integrated  Data  Systems  (IDS). 
V-Realm  products  have  been  transferred  to  the  Ligos  Corporation,  a  subsidiary  of  IDS.  They  no 
longer  appear  to  support  this  viewer  ,  but  they  do  sell  V-Realm  Builder,  a  package  for  creating 
VRML  worlds.  Several  tests  were  not  completed  due  to  the  discontinuation  of  this  viewer. 

The  V-Realm  viewer  is  available  as  a  stand-alone  and  a  plug-in  product  to  a  Web 
browser.  The  viewer  supports  only  Version  1.0  of  the  VRML  specification.  The  viewer  allows 
links  to  pictures  and  text  as  well  as  text  in  the  environment.  It  fully  supports  the  polygon  criteria, 
color  criteria,  and  most  of  the  features  of  shading  and  textures.  Additionally,  it  was  able  to 
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handle  compressed  files  and  supports  LOD,  limited  features  of  perspective  view,  groupings, 
mathematical  objects  and  wireframe. 

It  does  not  support  orthogonal  viewing,  lines,  the  undo  feature,  transparency,  multiple 
simultaneous  views,  measurement  units,  animation,  multiple  file  loads,  arbitrary  clip  planes, 
points,  or  light  sources. 
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3D  MODEL  CREATION 


Model  Creation  Process 

In  support  of  the  comparative  study,  a  3D  model  of  an  F-15  aircraft,  including  a  detailed 
model  of  bay  3R,  was  required.  An  initial  search  to  find  a  complete  model  of  an  F-15,  complete 
with  all  internal  parts.  Line  Replaceable  Units  (LRUs),  and  structural  elements  was  conducted. 
Although  several  labs  claimed  to  have  such  a  model,  or  parts  of  one,  most  would  not  or  could  not 
(due  to  classification)  release  the  data.  As  a  result,  the  focus  of  the  effort  turned  to  creating 
suitable  models. 

Due  to  its  large  user  base,  number  of  features,  and  support,  the  tool  chosen  for  the 
modeling  effort  was  3D  Studio  Max  vl.2.  The  modeling  effort  was  divided  into  five  sections: 

1)  Finding  and  converting  an  outer  shell  of  an  F-15  model  to  a  file  format  that  would  be 
usable  by  3D  Studio  Max. 

2)  Create  the  shelves  and  LRUs  of  bay  3R. 

3)  Combine  the  models  from  steps  1  and  2  to  create  a  single  F-15  model. 

4)  Add  animation  and  VRML  triggers  to  the  F-15  model. 

5)  Converting  the  F-15  model  to  MAiXTVRML  formats. 

Identifying  and  Converting  F-15  Outer  Shell 

The  DEPTH  (Design  Evaluation  for  Personnel,  Training,  and  Human  Factors)  laboratory, 
at  Wright  Patterson  AFB,  Dayton,  OH,  provided  a  model  of  an  F-15  which  had  previously  been 
used  for  several  studies  involving  simulation  of  maintenance  activity.  The  model  contained  a 
realistic  outer  shell  of  an  F-15  with  the  3R  bay  already  marked,  and  was  ideal  for  the  purposes  of 
this  project. 

However,  this  model  was  being  used  in  the  DEPTH  lab  under  Transom  Jack,  a  human¬ 
centric  visual  simulator.  In  order  to  use  the  F-15  model  for  this  project,  it  needed  to  be  converted 
from  Transom  Jack’s  internal  format  into  a  format  suitable  for  the  project’s  modeler  program,  3D 
Studio  Max.  At  the  time,  there  were  no  converters  capable  of  a  direct  conversion  from  the 
Transom  Jack  file  format  to  the  3D  Studio  Max  file  format. 
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By  using  several  different  converters  in  conjunction  with  one  another,  the  ability  to 
convert  files  from  Transom  Jack  to  3D  Studio  Max  was  created.  By  converting  from  Transom 
Jack’s  format  to  VRML  1.0  to  AutoCAD’s  DXF  and  finally  to  Kinetixs  MAX  file  format,  the 
model  data  was  loaded  into  3D  Studio  Max.  A  discussion  of  this  process  follows. 

1)  Convert  Transom  Jack’s  native  format  to  VRML  1.0: 

The  DEPTH  lab’s  version  of  Transom  Jack  contained  a  plug-in  module  which  allowed 
any  model  in  Transom  Jack  to  be  exported  directly  into  a  VRML  1.0  file.  However,  using 
the  plug-in  produced  VRML  1.0  files  with  several  syntax  errors,  all  of  which  were  easily 
fixed  by  editing  the  file  with  a  text  editor.  The  conversion  was  completed  on  a  Silicon 
Graphics  Onyx  computer. 

2)  Convert  VRML  1.0  into  DXF: 

It  should  be  noted  that  many  converters  exist  to  convert  a  format  to  VRML  1.0,  but  there 
are  very  few  which  converted /rom  VRML  1.0  to  another  3D  file  format.  After  much 
searching,  a  freeware  converter  named  Crossroads  (a  1 .0  beta,  created  by  Keith  Rule)  was 
found.  It  could  read  in  the  VRML  1.0  F- 15  model  on  a  Pentium  200  Windows  NT  system 
and  converted  to  a  DXF  file  with  no  visible  errors. 

3)  Convert  DXF  to  3D  Studio  Max’s  internal  format: 

Kinetix,  the  creator  of  3D  Studio  Max,  offered  a  free  plug-in  which  read  DXF  files  into 
3D  Studio  Max.  The  conversion  took  several  hours  of  solid  CPU  time  to  convert  the  data 
from  DXF  to  MAX  using  a  Pentium  200  MHz  under  Windows  NT . 

One  problem  with  this  method  of  successive  conversions  is  that  information  can  be  lost 
between  each  transfer.  For  example,  VRML  1.0  can  handle  some  limited  forms  of  animation  in 
its  format,  and  AutoCAD’s  DXF  cannot.  Therefore,  any  conversion  of  VRML  1.0  to  DXF  will 
result  in  a  model  without  animation  characteristics.  If  this  process  were  to  be  extended  (for 
example,  VRML  1.0  ->  DXF  ->  POV  ->  OBJ  ->  FLT),  more  and  more  characteristics  would  be 
lost  between  the  original  format  and  the  final  format  due  to  the  differing  function  sets  of  each  of 
the  formats. 
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There  was  also  the  issue  of  poor  converters  -  very  few  converters  actually  convert  the 
entire  feature  sets  of  the  3D  file  formats.  Although  both  VRML  1.0  and  DXF  can  store  models  as 
a  hierarchy  of  objects,  there  was  no  guarantee  that  a  particular  converter  will  actually  process  the 
hierarchy  feature.  Implementations  of  these  converters  vary  greatly,  and  the  difference  between  a 
good  converter  and  a  poor  one  is  often  whether  or  not  a  description  of  what  features  it  could  and 
could  not  handle  is  included. 

Neither  of  these  problems  presented  a  serious  barrier  to  the  modeling  effort,  since  all  that 
was  needed  was  the  set  of  polygonal  geometry  representing  the  F-15’s  outer  hull,  and  all 
converters  that  were  used  processed  this  feature. 

Figure  5  shows  the  F-15  shell  which  was  converted  and  eventually  imported  into  3D 
Studio  Max. 


Figure  5.  3D  Studio  Max  Rendering  of  F-15  Shell. 

Creating  Shelf  Assembly  and  LRUs  of  Bay  3R 

The  1F-15A-39  ABDR  TO  was  used  extensively  to  create  the  shelf  assembly  of  bay  3R. 
It  should  be  noted  that  although  the  TO  was  used  to  build  a  virtual  F-15  in  this  project,  it  was 
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never  intended  for  this  purpose.  Consequently,  many  measurement  calculations  for  various 
components  were  not  included  in  the  TO,  thus  requiring  the  software  team  to  estimate 
measurements  directly  from  orthogonal  view  diagrams.  These  measurements  were  then  used  to 
model  the  3D  objects  representing  the  shelves,  centerline  plenum,  and  several  of  the 
miscellaneous  features  such  as  wire  holes  and  ribbings  on  the  shelves.  The  models  were  fairly 
simple,  and  required  only  setting  basic  color  and  lighting  components  similar  to  the  real-life 
shelves.  All  colors  chosen  (generally  a  metallic  green-blue  for  the  shelves)  were  based  on  the 
colors  found  in  a  real  F-15. 

Although  the  TO  contained  detailed  information  on  the  behavior  of  damaged  LRUs  and  a 
single  black  and  white  drawing  of  the  3R  bay,  more  information  was  required  to  accurately 
model  the  LRUs  for  the  3D  model.  To  obtain  this  information,  F-15  3R  bays  were  photographed 
at  the  Air  Force  Museum  at  Wright-Patterson  AFB  and  the  653  CLSS  at  Robins  AFB. 

These  photos  provided  rough  measurements  of  height  and  width  for  each  LRU. 
Simulated  objects  representing  the  LRUs  were  modeled  under  3D  Studio  Max.  The  models  were 
slightly  more  complex  than  the  shelves  and  included  texture  maps  as  well  as  some  extra  details. 
By  editing  the  digital  photographs  with  a  2D  paint  program  (Paint  Shop  Pro  5.0),  texture  maps 
were  created  for  the  front  faces  of  each  of  the  LRUs.  These  texture  maps  were  then  applied  to  the 
3D  objects,  and  a  reasonable  facsimile  of  the  LRUs  was  created.  Figure  6  shows  the  model  of 
bay  3R.  Care  was  taken  to  ensure  that  the  final  texture  maps  kept  the  same  color  scheme  as  the 
original  items. 


65 


Figure  6.  Bay  3R. 


Combining  the  F-15  Shell  and  Bay  3R 

Once  both  models  were  created,  it  was  necessary  to  combine  the  F-15  shell  and  the  LRU 
models/shelves  to  create  a  single  model.  Despite  the  (often)  rough  measurements  used  to  create 
the  bay  3R  shelves,  the  entire  shelf  assembly  fit  into  the  F-15  shell  provided  by  the  DEPTH  lab 
with  minor  sizing  modifications. 


Some  optimization  of  the  combined  model  was  performed  at  this  stage  to  improve  the 
frame  rate.  Several  surfaces,  especially  parts  of  the  F-15  shell,  contained  far  more  detail  than  was 
needed  for  the  project.  As  a  result,  these  parts,  including  the  landing  gear  and  engine  nozzle 
sections,  had  their  total  number  of  polygons  reduced  by  nearly  one-third  to  increase  the  frame 
rate.  Several  minor  flaws  in  the  model,  such  as  polygons  with  reversed  backfacing,  were  also 
cleaned  up  during  this  stage.  Figure  7  shows  the  results  of  combining  the  F-15  shell  with  Bay 
3R. 


Figure  7.  Complete  FI 5  Model. 

Creating  Animation  and  Triggers 

A  small  three-button  panel  was  created  under  each  LRU  in  the  model.  VRML  2.0 
TouchSensor  and  TimeSensor  triggers  were  added  to  each  button  in  the  panel.  These  triggers, 
when  used  in  conjunction  with  animation  capabilities,  allowed  the  user  to  change  whether  or  not 
a  LRU  was  displayed  on-screen  and  with  or  without  textual  information. 

This  (label)  textural  information  was  created  to  help  in  the  identification  and  location  of 
components  within  a  graphic.  System  assessment/repair  text  from  the  Automatic  Flight  Control 
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System,  Communication,  and  Navigation  Systems  of  the  1F-15A-39  ABDR  TO  was  converted 
into  a  set  of  HTML  files.  By  linking  TouchSensor  triggers  to  the  HTML  documents,  the  ability 
to  spawn  additional  browser  windows  containing  this  textural  information  was  created.  It  should 
be  noted  that  adding  the  VRML  triggers  and  HTML  files  only  produced  these  extra  capabilities 
in  a  VRML  simulation,  specifically,  Cosmo  Player.  For  the  3D  Studio  Max  simulation, 
equivalent  functionalities  were  found  in  Max’s  features  -  specifically,  the  onscreen  display/ 
hiding  capability  was  created  with  the  Unhide  /  Hide  By  Name  feature,  and  the  display 
information  capability  was  created  by  adding  system  assessment  text  into  a  spawnable  Properties 
window. 

Finally,  the  models  were  exported  from  3D  Studio  Max’s  internal  format  to  VRML  2.0 
for  use  in  the  VRML  browsers.  The  original  MAX  file  was  used  during  the  review  of  3D  Studio 
Max  as  a  3D  viewer. 

Converting  the  F-15  Model  to  MAX/VRML  Formats 

Based  on  the  evaluation  results  of  the  3D  viewers,  Cosmo  Player  and  3D  Studio  Max 
were  selected  for  use  in  the  execution  of  the  comparative  study.  The  required  file  formats  for 
Cosmo  Player  and  3D  Studio  Max  are  VRML  2.0  and  MAX,  respectively.  Since  the  model  was 
created  using  3D  Studio  Max,  it  was  automatically  available  in  the  MAX  file  format.  The  export 
feature  of  3D  Studio  was  used  to  produce  a  VRML  2.0  version  of  the  model 

Cost  of  Model  Creation 

Table  9  summarizes  the  time  and  cost  for  developing  the  above  described  F15  model. 
The  table  breaks  up  the  creation  and  development  of  the  model  into  five  main  tasks.  Associated 
with  each  of  the  tasks  is  the  time  required  to  complete  the  tasks.  Finally,  to  compute  the  cost 
associated  with  each  task  a  labor  rate  of  $65.49  an  hour  was  used.  This  labor  rate  is  based  on 
TASC’s  GSA  ADP  labor  rate  for  a  software  developer  with  two  years  experience. 
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Table  9.  Cost  of  Model  Creation 


'['Task 

Time  ' --'rT 

Cost 

Identify  and  Converting  F-15  Model 

32  hours 

$2,096 

Model  Shelves,  centerline  plenum,  and  LRUs 

38  hours 

$2,489 

Create  and  add  texture  maps  to  LRUs 

21  hours 

$1,375 

Combine  shelves  and  LRUs  in  F-15  shell 

5  hours 

$327 

Develop  VRML  animation 

32  hours 

$2,096 

Totals 

128  hours 

$8,383 
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COMPARATIVE  STUDY 


Method 

A  comparative  study  was  conducted  on  the  use  of  3D  graphics  in  F-15  maintenance.  This 
portion  of  the  project  focused  on  the  direct  application  of  a  3D  model  for  use  in  a  particular 
maintenance  task.  The  findings  from  the  CTA  were  used  to  help  design  a  3D  graphic  (model)  of 
bay  3R  of  the  F-15.  Two  viewers,  Cosmo  Player,  from  here  on  referred  to  as  Cosmo,  and  3D 
Studio  Max,  were  used  to  display  and  manipulate  the  3D  model. 

Subjects 

Six  maintainers  participated  in  the  comparative  study.  None  of  these  maintainers 
participated  in  the  CTA.  All  had  ABDR  maintenance  experience.  They  also  had  an  average  of 
seven  years  computer  experience.  Three  had  experience  with  3D  graphics,  but  none  had  any 
familiarity  with  the  test  bed  viewing  applications. 

Procedure 

Before  starting,  the  maintainers  were  briefed  on  the  background  and  goals  of  this  portion 
of  the  project.  After  the  briefing,  the  maintainers  completed  a  background  information 
questionnaire  (Appendix  D).  They  were  then  given  only  one  of  the  two  viewers  to  evaluate. 
Maintainers  that  participated  the  first  day  of  data  collection  evaluated  the  Cosmo  viewer  while 
the  second  day’s  maintainers  used  3D  Studio  Max.  Each  maintainer  was  given  a  choice  of  input 
device:  an  external  mouse  or  an  integrated  pointing  stick  controller  located  on  the  keyboard. 
Because  of  the  maintainers’  familiarity  with  the  external  mouse,  all  selected  this  means  of 
control. 


Since  the  maintainers  were  not  familiar  with  either  viewer,  they  were  given  extensive 
instructions.  The  maintainers  were  first  shown  all  of  the  available  features  in  the  particular 
viewer.  After  this,  they  were  instructed  to  perform  several  tasks  that  utilized  each  of  these 
features.  Once  the  maintainers  and  the  experimenter  felt  confident  that  the  task  could  be 
completed,  training  was  terminated. 
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The  main  study  focused  on  an  F- 15  aircraft.  The  aircraft  used  was  located  in  a  hanger  at 
Robins  AFB.  A  table  and  laptop  computer  were  placed  at  the  nose  end  of  the  plane.  The  3D 
model  and  viewers  were  run  on  a  Pentium  233  MHz  laptop  PC  with  a  screen  size  of  13.3  inches. 

The  display  signal  from  the  laptop  was  repeated  on  an  additional  monitor,  which  allowed 
researchers  the  opportunity  to  watch  and  videotape  the  use  of  the  model  and  viewer.  The  post¬ 
task  questionnaire  presented  to  the  maintainers  was  audiotaped  for  use  in  analysis.  The 
questionnaire  asked  a  series  of  probing  questions  to  gather  information  on  the  3D  model  and 
viewer  (Appendix  E). 

To  start  the  task,  the  maintainers  were  given  a  hypothetical  scenario  describing  the 
damage  sustained  by  the  aircraft.  They  were  asked  to  identify  all  of  the  aircraft  damage,  signified 
by  white  stickers  placed  on  the  aircraft,  and  to  mark  it  on  the  3D-computer  model.  There  were  a 
total  of  nine  damage  locations.  The  order  in  which  the  maintainers  located  and  marked  the 
damage  on  the  model  was  unrestricted.  All  of  the  maintainers  alternated  between  the  aircraft  and 
the  3D  model  to  correctly  identify  all  nine  locations  of  damage.  A  copy  of  the  protocol  for  the 
comparative  study  can  be  found  in  Appendix  F. 

Viewers 


Cosmo  Player  2.1 

The  Cosmo  Player  version  2.1  (Cosmo)  is  a  Web  browser  plug-in  which  extends  the 
functionality  of  both  Netscape's  Communicator  4.x  version  and  Microsoft  s  Internet  Explorer 
4.x  version  browsers.  For  this  study,  Netscape’s  Communicator  version  4.04  was  used.  The 
Cosmo  plug-in  provides  a  viewer  for  3D  models  authored  in  the  VRML.  A  dashboard  of  controls 
at  the  bottom  of  the  screen  is  provided  in  Cosmo  for  manipulating  the  3D  model.  In  this  study, 
only  a  subset  of  the  available  controls  was  made  available  to  the  maintainers:  zoom,  rotate,  pan, 
tilt,  go,  slide,  seek,  straighten,  and  undo/redo.  Three  other  capabilities  which  were  not  built  in 
features  of  Cosmo  were  also  available  to  the  maintainers:  information,  remove/replace,  and 
mark.  It  should  be  noted  that  all  of  the  controls  available  in  Cosmo  can  be  executed  using  the 
keyboard,  but  for  this  study,  the  maintainers  were  instructed  to  use  the  pointing  device.  A  short 
description  of  each  of  these  controls  follows. 
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Cosmo  Player  2.1  Features: 


Zoom.  Allows  the  user  to  increase  or  decrease  the  effective  size  of  the  3D  model. 

Rotate.  Allows  the  user  to  spin  (rotate)  the  3D  model  about  any  axis.  The  axis  of  rotation  is 
determined  by  the  direction  the  user  drags  the  mouse.  For  example,  if  the  user  drags  the  mouse 
horizontally  left  to  right  directions,  the  model  will  be  rotated  around  the  vertical  axis  in  a  left  to 
right  direction.  Similarly,  if  the  user  drags  the  mouse  in  a  vertical  direction,  the  model  will  be 
rotated  around  the  horizontal  axis.  Other  angles  of  rotation  can  be  accomplished  by  dragging  the 
mouse  in  an  angular  direction. 

Pan.  Allows  the  user  to  move  the  model  left,  right,  up,  or  down.  Dragging  the  mouse  to  the  left 
moves  the  model  to  the  left,  etc.  The  speed  at  which  the  model  moves  using  the  “pan”  feature  is 
constant. 

Go.  Allows  the  user  to  move  in  any  direction  on  the  screen.  This  feature  turns  the  user's  point  of 
view  in  the  direction  of  the  movement.  Dragging  up  moves  the  user’s  view  closer,  dragging 
down  moves  the  user’s  view  outward,  dragging  left  or  right  moves  the  user’s  view  to  the  left  or 
right. 

Slide.  Like  “pan”,  the  “slide”  feature  allows  the  user  to  move  the  model  left,  right,  up,  or  down. 
The  main  difference  between  the  “slide”  and  “pan”  features  is  that  the  speed  at  which  the  model 
moves  can  be  controlled  using  the  “slide”  feature  while  it  is  fixed  with  the  “pan”  feature.  The 
slide  feature  does  not  turn  the  user's  point  of  view  (as  does  the  Go  feature);  it  simply  moves  the 
model  in  the  desired  direction  (the  user’s  point  of  view  is  fixed). 

Tilt.  Allows  the  user’s  point  of  view  to  be  tilted.  This  can  be  thought  as  tilting  a  camera  along  a 
fixed  axis  of  rotation.  The  “tilt”  feature  will  not  allow  the  user  to  turn  the  view  upside  down. 

Seek.  Allows  the  user  quick  navigation  to  an  area  within  the  3D  model.  Once  the  seek  button  is 
selected,  the  user  places  the  cursor  on  an  area  of  interest  and  clicks  the  left  mouse  button.  Cosmo 
will  then  position  the  user’s  point  of  view  at  the  area  of  interest. 

Straighten.  Returns  the  user’s  point  of  view  to  a  straight  and  level  position. 
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Undo/Redo.  Cosmo  maintains  a  history  of  moves  that  the  user  has  executed.  The  “undo”  feature 
allows  the  user  to  undo  the  last  operation  executed.  The  “undo”  feature  can  be  used  in  succession 
to  undo  the  last  several  operations.  “Redo”  allows  the  user  to  repeat  the  last  operation  executed. 

Custom  Features: 

Information.  The  information  feature  allowed  the  user  to  access  textual  information  associated 
with  a  particular  LRU.  This  feature  was  implemented  by  utilizing  Touch  Sensors  in  the  VRML 
language,  which  provided  the  ability  to  set  up  “hot  spots”  in  the  3D  model.  Associated  with  these 
“hot  spots”  was  an  event  that  loaded  an  HTML  page  containing  information  about  the  selected 

LRU. 

Remove/Replace.  Again  using  “hot  spots,”  the  ability  to  remove  and  replace  objects  in  the  3D 
model  was  developed.  Each  of  the  LRUs  in  the  model  had  three  associated  buttons  (remove, 
replace,  and  information)  which  when  selected  with  the  mouse  would  remove  the  object  (if  not 
already  removed)  or  replace  the  object  (if  already  removed). 

Mark.  The  “mark”  feature  provided  the  maintainer  with  the  ability  to  mark  the  simulated 
damage  on  the  3D  model.  A  screen  capture  program  called  HyperSnap-DX  Pro  was  used  to 
implement  the  “mark”  feature.  To  mark  a  damage  spot  on  the  model,  the  maintainer  placed  the 
cursor  at  the  desired  location  and  selected  a  defined  essential  combination.  Which  triggered  the 
HyperSnap  application  to  execute  a  screen  capture,  which  recorded  the  cursor  location  at  the 
damaged  spot  on  the  screen  and  associated  with  the  aircraft. 

3D  Studio  Max  1 .2 

The  other  viewer  used  in  the  study  was  a  3D  modeling  package  called  3D  Studio  Max. 
As  opposed  to  the  plug-in  Cosmo,  3D  Studio  Max  is  a  standalone  package  designed  for  creating 
and  viewing  3D  models.  3D  Studio  Max  provides  for  many  different  possibilities  to  view  a 
model.  In  this  study,  the  team  used  a  single,  fully  rendered  perspective  view.  This  view  was 
analogous  to  the  view  provided  in  Cosmo.  Because  it  is  used  primarily  as  a  design  tool  (as 
opposed  to  being  strictly  a  viewer),  3D  Studio  Max  contains  many  controls  and  features.  In  order 
to  reduce  complexity,  the  maintainers  used  only  a  small  subset  of  the  available  features:  zoom, 
pan,  seek,  rotate  remove/replace,  and  undo/redo.  Additional  features  demonstrated  to  but  not 
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used  by  the  maintainers.  These  included  multiple  simultaneous  views,  measurement  units,  and 
wire  frame  view.  A  brief  explanation  of  each  of  the  features  is  provided  below. 

3D  Studio  Max  Features  Used  bv  Maintainers: 

Zoom.  Allows  the  user  to  increase  or  decrease  the  effective  size  of  the  model. 

Pan:  Allows  the  user  to  move  the  model  left,  right,  up,  or  down.  Dragging  the  mouse  to  the  left 
moves  the  model  to  the  right,  dragging  the  mouse  to  the  right  moves  the  model  to  the  left,  etc. 

Seek.  A  two  step  process  had  to  be  followed  to  achieve  seek  feature.  The  first  step  consisted  of 
selecting  the  desired  object.  After  selecting  an  object,  the  user  then  selected  the  “zoom  extents” 
command,  which  zoomed  the  viewer  on  the  selected  object. 

Rotate.  Allows  the  user  to  rotate  the  model  along  any  axis.  As  with  Cosmo,  the  direction  in 
which  the  user  drags  the  mouse  determines  the  angle  of  rotation.  Another  aspect  of  the  rotate 
feature  in  3D  Studio  Max  is  the  ability  to  constrain  the  axis  of  rotation  along  the  horizontal 
(typically  the  x-axis)  or  the  vertical  axis  (typically  the  z-axis). 

Remove/replace.  Like  the  seek  feature  in  3D  Studio  Max,  the  “remove/replace”  feature 
required  a  two  step  process.  To  remove  an  object,  the  first  step  was  to  select  the  object.  To 
complete  the  procedure,  the  “hide  selected”  option  was  chosen  and  the  object  was  removed.  For 
the  “replace”  feature,  the  object  was  selected  by  name  from  a  list  of  hidden  objects.  The  “unhide 
selected”  option  was  chosen  and  the  object  reappeared  in  the  model. 

Undo/redo.  This  feature  was  identical  to  the  “undo/redo”  feature  found  in  Cosmo. 

3D  Studio  Max  Features  Demonstrated  to  Maintainers: 

Multiple  simultaneous  views.  The  “multiple  simultaneous  views”  feature  found  in  3D  Studio 
Max  allows  the  user  to  have  up  to  four  different  views  of  the  model  on  the  screen  at  one  time. 
For  example,  a  top,  front,  side,  and  perspective  view  can  all  be  shown. 
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Measurement  units.  3D  Studio  Max  allows  the  user  to  set  up  measurement  units  for  a  model. 
Once  the  measurement  units  have  been  established  and  entered  by  the  user,  the  user  can 
determine  the  x,y,z  coordinates  of  the  mouse  relative  on  the  measurement  units  supplied. 

Wireframe  view.  3D  Studio  Max  can  be  configured  so  that  the  program  switches  to  a  wireframe 
mode  when  the  model  is  being  manipulated  (rotated,  zoomed,  etc.).  This  has  the  advantage  of 
providing  a  much  faster  frame  rate  (update  speed)  than  under  normal  conditions. 

Data  Analysis 

The  raw  data  consisted  of  the  research  team’s  notes,  audio  recordings  of  the  prebriefing 
and  postbriefing,  video  recordings  of  the  scenario  execution,  and  the  results  of  the 
questionnaires.  The  answers  from  the  questionnaire  are  summarized  below.  After  a  series  of 
general  questions,  two  questions  were  asked  about  each  of  the  operations  available  in  the 
viewers.  A  table  was  created  for  each  feature  to  summarize  the  maintainers’  responses.  An 
example  of  one  of  the  tables  is  provided  below.  Columns  one  through  five  identify  the  subject, 
ease  of  use,  number  of  operations,  percentage  of  what  an  operation  was  used,  and  the  operation  s 
rating.  The  following  section  summarizes  the  subjects’  answers  for  each  of  the  questions  in  the 
questionnaire.  (Note:  Some  of  the  maintainers  that  used  the  Cosmo  viewer  used  the  straighten 
feature,  which  was  not  addressed  in  the  questionnaire.) 


Viewer:  Cosmo  Operation:  Zoom 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  1 

Highly  Satisfactory 

1 

2% 

Useful 

Subject  2 

Highly  Satisfactory 

8 

13% 

Useful 

Subject  3 

Highly  Satisfactory 

1 

2% 

Essential 

Two  sets  of  scales  were  used  to  evaluate  the  features.  When  asked  to  rank  the  ease  of  use 
for  the  features  the  maintainers  were  given  a  scale  ranging  between  “Outstanding”  and 
“Unsatisfactory”  as  seen  below. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 
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In  some  particular  cases  they  were  also  asked  to  rate  the  necessity  of  the  feature  between 
“Essential”  and  “Useless”  as  seen  below. 


Essential  Useful  Useless 


QUESTION;  Were  the  3D  graphics  easy  to  use? 

All  but  one  of  the  maintainers  stated  the  3D  graphics  were  easy  to  use.  The  one 
maintainer  who  responded  “No”  commented  that  with  sufficient  practice  the  graphics  would  be 
easy  to  use. 

QUESTION;  Did  the  laptop  computer  hinder  your  ability  to  use  the  3D  graphics? 

Four  of  the  six  maintainers  responded  that  the  laptop  did  not  hinder  their  ability  to  use  the 
3D  graphics.  One  of  maintainers,  who  indicated  that  the  laptop  hindered  his  ability,  also  stated 
that  the  computer  was  slow.  The  same  maintainer  commented  that  even  if  the  laptop  was  slow, 
maintainers  would  still  use  it.  Another  maintainer  stated,  the  ability  to  take  the  laptop  to  aircraft 
would  be  very  beneficial  because  it  would  be  easily  accessible  while  doing  an  assessment  . 

Two  issues  were  raised  regarding  the  laptop’s  use  in  an  operational  environment.  First, 
one  subject  expressed  a  concern  about  the  survivability  and  reliability  of  a  laptop  computer  at  a 
deployed  site.  Environmental  conditions  can  be  extreme  during  a  deployment,  jeopardizing  the 
laptop’s  reliability. 

Second,  the  use  of  electronic  devices  near  the  fuel  cells  was  also  discussed.  An  electronic 
component  may  not  be  permitted  when  performing  assessments  near  fuel  cells,  due  to  the 
possibility  of  igniting  the  vapors. 

QUESTIQN;  Adequacy  of  the  screen  size  for  displaying  the  3D  graphics. 

All  of  the  maintainers  ranked  the  adequacy  of  the  screen  size  as  satisfactory  or  better  with 
one  of  the  maintainers  ranking  it  “Outstanding.”  One  of  the  maintainers  mentioned  that  the  larger 


the  screen  size  the  better. 


QUESTION;  Adequacy  of  the  input  device  for  manipulating  the  3D  graphics. 

All  of  the  maintainers  ranked  the  adequacy  of  the  input  device  to  be  between  “Highly 
Satisfactory”  and  “Satisfactory.”  Three  of  the  maintainers  mentioned  the  possibility  of  using  a 
trackball  or  joystick  as  an  alternative  input  device.  Another  comment  was  made  in  regard  to 
having  to  do  assessments  in  full  chemical  gear  including  heavy  rubber  gloves.  This  maintainer 
said  that  it  would  be  very  difficult  to  use  a  pointing  stick  or  touch  pad  (typical  integrated  input 
devices  in  laptop  computers)  while  wearing  the  gloves. 

QUESTION:  Use  of  the  laptop  computer  as  a  tool  to  manipulate  the  3D  graphics. 

Five  out  of  the  six  maintainers  ranked  the  use  of  the  laptop  computer  to  manipulate  the 
3D  graphics  as  “Highly  Satisfactory”  or  above.  The  remaining  maintainer  gave  a  ranking 
between  “Highly  Satisfactory”  to  “Satisfactory”  and  commented  that  he  would  have  preferred  a 
faster  laptop,  which  would  have  updated  the  graphics  more  quickly. 

QUESTION:  Were  the  3D  graphics  displayed  in  this  task  realistic  and  reliable?  Why? 

Four  of  the  maintainers  stated  that  the  graphics  were  realistic  and  reliable.  Three  of  the 
maintainers  commented  that  there  needed  to  be  more  detail  in  the  model  if  the  system  was  ever 
to  go  into  production.  Two  of  the  maintainers  commented  that  more  detail  on  the  structure 
should  be  modeled.  Another  maintainer  pointed  out  that  the  hose  clamps  that  secured  the  coolant 
line  to  the  center  plenum  were  not  modeled  which  caused  him  some  problems  in  accurately 
identifying  one  of  the  damaged  areas. 

QUESTIQN;  Realism  of  the  3D  graphics. 

Three  of  the  maintainers  commented  that  the  realism  of  the  graphics  was  outstanding  and 
the  remaining  three  gave  a  ranking  between  “Highly  Satisfactory”  to  “Satisfactory.  One  of  the 
maintainers  commented,  “It  made  you  feel  that  you  were  actually  on  the  aircraft. 

QUESTIQN:  Were  there  any  misplaced  or  missing  landmarks  in  the  3D  graphics?  If  there 
were,  did  they  cause  problems  in  the  scenario? 
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All  six  of  the  maintainers  cited  misplaced  or  missing  landmarks  in  the  model.  Three  of 
the  maintainers  mentioned  missing  details  in  the  modeling  of  the  structure.  Some  of  the  specific 
areas  mentioned  were  the  lightening  holes  were  not  modeled  and  there  was  a  rib  that  connected 
two  of  the  shelves  that  was  missing.  One  maintainer  recommended  more  detail  be  modeled  on 
the  structure.  He  said  that  the  structure  of  the  aircraft  rarely  changes.  He  also  pointed  out  that  in 
an  ABDR  assessment,  if  an  LRU  is  determined  to  be  damaged,  it  is  removed  and  replaced  with  a 
new  one.  Because  of  this,  there  does  not  need  to  be  a  great  deal  of  detail  in  modeling  the  LRUs. 

Another  maintainer  pointed  out  that  the  hose  clamps  that  attached  the  coolant  line  to  the 
center  plenum  were  missing  and  caused  him  difficulty  in  properly  locating  some  of  the  damaged 
areas.  This  same  maintainer  mentioned  that  it  would  be  helpful  if  the  tubes  were  labeled  as  they 
are  on  the  aircraft  (high  pressure,  low  pressure,  etc.). 

QUESTION;  Could  you  please  provide  examples  of  additional  manipulation  features  that 
would  be  helpful? 

Five  out  of  the  six  maintainers  recommended  additional  manipulation  features.  One  of 
the  maintainers  suggested  providing  the  capabilities  to  enter  coordinates  and  has  the  viewer 
position  the  model  to  the  entered  coordinates.  The  same  maintainer  also  suggested  allowing  the 
user  to  enter  an  area  of  damage,  for  example  bay  3R,  and  have  the  viewer  position  the  model  so 
that  bay  was  visible. 

Another  suggestion  was  to  use  a  trackball  and  directional  buttons  for  navigation.  This 
maintainer  suggested  providing  controls  similar  to  a  video  game  for  manipulating  the  model.  The 
directional  buttons  would  allow  constrained  manipulation  along  a  single  axis.  Another 
suggestion  was  to  use  a  touch  screen  to  manipulate  the  image. 
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QUESTION;  Please  mark  the  features  you  used  in  the  task  just  performed. 


Sub.  1 

Sub.  2 

Sub.  3 

Sub.  4 

Sub.  5 

Sub.  6 

Seek 

X 

X 

X 

X 

X 

X 

Zoom 

X 

X 

X 

X 

X 

X 

Rotate 

X 

X 

X 

X 

X 

X 

Pan 

X 

X 

X 

X 

X 

Undo/Redo 

X 

X 

X 

X 

X 

X 

Remove/Replace 

X 

X 

X 

X 

X 

X 

Mark 

X 

X 

X 

X 

X 

X 

Information 

X 

X 

Tilt 

X 

X 

X 

Go 

X 

X 

X 

Slide 

X 

X 

QUESTION;  Ease  in  using  the  seek  feature.  Please  rate  the  seek  feature. 


Viewer:  Cosmo  Feature:  Seek 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subiect  1 

Outstanding 

4 

6% 

Essential 

Subject  2 

Outstanding 

12 

19% 

Essential 

Subject  3 

Satisfactory 

1 

2%  ^ 

Useful 

3D  Studio  Max  Feature:  Seek 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subiect  4 

Satisfactory 

4 

8% 

Essential 

Subject  5 

Outstanding 

4 

8% 

Essential 

Subject  6 

Outstanding 

4 

6%  n 

Useful 

QUESTION;  Ease  in  using  the  zoom  feature.  Please  rate  the  zoom  feature. 


Viewer:  Cosmo  Feature:  Zoom 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subiect  1 

Highly  Satisfactory 

1 

2% 

Useful 

Subject  2 

Highly  Satisfactory 

8 

13% 

Useful 

Subject  3 

Highly  Satisfactory 

1 

2% 

Essential 

Viewer:  3D  Studio  Max  Feature:  Zoom 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subiect  4 

Marginal 

9 

17% 

Essential 

Subject  5 

Highly  Satisfactory 

12 

23% 

Essential 

Subject  6 

Highly  Satisfactory 

13 

21%  n 

Essential 
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UESTION;  Ease  in  using  the  rotate  feature.  Please  rate  the  rotate  feature. 


V fewer:  Cosmo _ Feature:  Rotate 


Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subiect  1 

Highly  Satisfactory 

7 

11% 

Useful 

Subiect  2 

Highly  Satisfactory 

12 

19% 

Essential 

Subject  3 

Highly  Satisfactory 

5 

11% 

Essential 

Viewer:  3D  Studio  Max 

Feature:  Rotate 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  4 

Marginal 

9 

17% 

Essential 

Subject  5 

Highly  Satisfactory 

12 

23% 

Essential 

Subject  6 

Marginal 

15 

24% 

Essential 

lUESTION;  Ease  in  using  the  pan  feature.  Please  rate  the  pan  feature. 


Viewer:  Cosmo _ Feature:  Pan 


Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subiect  1 

Highly  Satisfactory 

6 

10% 

Useful 

Subject  2 

Highly  Satisfactory 

7 

11% 

Essential 

Subject  3 

Did  Not  Use 

0 

0% 

N.A. 

1  Viewer:  3D  Studio  Max  Feature:  Pan 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  4 

Highly  Satisfactory 

8 

15% 

Essential 

Subject  5 

Satisfactory 

10 

19% 

Useful 

Subject  6 

Satisfactory 

14 

23% 

Essential 

QUESTION;  Ease  in  using  the  tilt  feature.  Please  rate  the  tilt  feature. 


Vif^wer:  Cosmo  Feature:  Tilt 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  1 

Highly  Satisfactory 

4 

6% 

Useful 

Subject  2 

Satisfactory 

3 

5% 

Useful 

Subject  3 

Highly  Satisfactory 

3 

7% 

Essential 

QUESTION:  Ease  in  using  the  go  feature.  Please  rate  the  go  feature. 


Viftwfir:  Cosmo  Feature:  Go 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  1 

Highly  Satisfactory 

1 

2%  1 

Useful 

Subject  2 

Satisfactory 

2 

3% 

Essential 

Subject  3 

Highly  Satisfactory 

7 

16% 

Essential 

QUESTION:  Ease  in  using  the  slide  feature.  Please  rate  the  slide  feature. 


Vipwer:  Cosmo  Feature:  Slide  . . 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  1 

Highly  Satisfactory 

6 

10% 

Useful 

Subject  2 

Did  Not  Use 

0 

0% 

N.A. 

Subject  3 

Highly  Satisfactory  ' 

4 

9% 

Essential 

()TTF.STTQN:  Ease  in  using  the  undo/redo  feature.  Please  rate  the  undo/redo  feature. 


Vipwer:  Cosmo  Feature:  Undo/Redo 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  1 

Highly  Satisfactory 

10 

16% 

Useful 

Subject  2 

Outstanding 

4 

6% 

Essential 

Subject  3 

Highly  Satisfactory 

6  1 

13% 

Useful 

Viewer:  3D  Studio  Max 

Feature:  Undo/Redo 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  4 

Marginal 

10 

19% 

Essential 

Subject  5 

Highly  Satisfactory 

1 

2% 

Essential 

Subject  6 

Outstanding 

6 

10% 

Essential 
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QUESTION;  Ease  in  using  the  remove/replace  feature.  Please  rate  the  remove/replace 
feature. 


I  Viewer:  Cosmo 

Feature:  Remove/Replace 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  1 

Highly  Satisfactory 

10 

16% 

Useful 

Subject  2 

Highly  Satisfactory 

5 

8% 

Essential 

Subject  3 

Highly  Satisfactory 

7 

16% 

Essential 

Viewer:  3D  Studio  Max 

Feature:  Remove/Replace 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  4 

Highly  Satisfactory 

5 

9% 

Essential 

Subject  5 

Highly  Satisfactory 

5 

10% 

Essential 

Subject  6 

Outstanding 

2 

3% 

Essential 

QUESTION:  Ease  in  using  the  mark  feature.  Please  rank  the  mark  feature. 


Viewer:  Cosmo 

Feature:  Mark 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  1 

Highly  Satisfactory 

8 

13% 

Useful 

Subject  2 

Highly  Satisfactory 

6 

9% 

Essential 

Subject  3 

Highly  Satisfactory 

7 

16% 

Essential 

Viewer:  3D  Studio  Max 

Feature:  Mark 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  4 

Highly  Satisfactory 

8 

15% 

Essential 

8 

15% 

Essential 

Outstanding 

8 

13% 

Essential 

QUESTION;  Ease  in  using  the  information  feature.  Please  rate  the  information  feature. 


Viewer:  Cosmo 

Feature:  Information 

Ease  of  Use 

No.  Ops 

%  Ops 

Rating 

Subject  1 

Outstanding 

3 

5% 

Essential 

Subject  2 

Highly  Satisfactory 

0 

Essential 

Subject  3 

Outstanding 

1 

2% 

Essential 
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QUESTION:  Would  you  have  preferred  a  paper-based  engineering  drawing  as  opposed  to 
a  3D  graphic  like  the  one  you  just  manipulated  in  Cosmo/3D  Studio  Max?  If 
so  when? 

There  were  three  “no”  responses,  two  “yes”  responses,  and  one  “depends”  response  to 
this  question.  There  was  only  one  maintainer  who  felt  strongly  he  would  prefer  paper  versus  a 
fully  detailed  3D  model.  A  common  comment  from  the  maintainers  was  that  a  greater  level  of 
detail  would  have  to  be  modeled  for  the  3D  graphics  to  be  useful.  One  of  the  maintainers  said 
that  if  there  was  a  greater  level  of  detail  and  if  the  model  provided  the  coordinates  of  the  mouse 
pointer  it  would  greatly  reduce  the  time  in  making  an  assessment.  Another  maintainer  mentioned 
that  the  system  needed  to  somehow  be  integrated  with  an  electronic  version  of  a  TO  in  order  for 
it  to  be  beneficial. 

QUESTION:  Could  you  complete  the  task  just  performed  with  multiple  paper-based 
drawings  or  multiple  pictures?  If  Yes,  how  many  views  would  you  need? 
Would  you  need  views  with  components  removed? 

Five  out  of  the  six  maintainers  responded  that  they  would  be  able  to  complete  the  task 
with  multiple  drawings.  All  maintainers  agreed  that  drawings  with  components  removed  would 
have  been  necessary  to  adequately  complete  the  task. 

QUESTION:  Would  you  prefer  to  use  multiple  fixed  perspective  paper  based  drawings  as 
opposed  to  the  3D  graphic  you  just  used? 

All  six  of  the  maintainers  responded  that  they  would  prefer  to  use  the  3D  graphic  model 
over  using  multiple  fixed  perspective  paper  drawings.  One  of  the  maintainers  commented  that  it 
would  be  much  easier  to  plot  the  path  of  the  damage  on  the  3D  model  than  on  paper  drawings. 
Another  point  was  they  typically  have  to  consult  multiple  TOs  in  order  to  do  an  assessment.  If  a 
complete  fully  detailed  3D  model  was  developed,  the  entire  assessment  could  be  accomplished 
more  quickly'  using  the  model  and  viewer  than  the  current  process  of  consulting  multiple  paper 
TOs. 
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QUESTION:  Ease  in  manipulation  within  Cosmo/3D  Studio  Max? 


Cosmo 

Ease  of  Manipulation 

Subject  1 

Outstanding 

Subject  2 

Highly  Satisfactory 

Subject  3 

Satisfactory 

3D  Studio  Max 

Ease  of  Manipulation 

Subject  4 

Highly  Satisfactory 

Subject  5 

Highly  Satisfactory 

Subject  6 

Highly  Satisfactory 

QUESTION:  Would  color  be  helpful  in  the  use  of  the  3D  graphics?  Why? 

All  six  of  the  maintainers  responded  that  color  would  be  helpful  in  the  use  of  3D 
graphics.  Three  of  the  maintainers  thought  that  color  should  be  used  to  help  identify  aspects  of 
the  plane  such  as  structure  categories  and  dangerous  areas.  The  structural  members  of  the  aircraft 
are  categorized  as  1-5  depending  on  their  criticality.  It  was  suggested  that  a  color  scheme  be 
implemented  that  would  allow  each  of  the  five  structural  categories  to  be  easily  identified  in  the 
model.  For  example,  in  the  3D  model,  identify  all  category  1  structural  members  using  the  color 
green  and  identify  all  of  the  category  2  structural  members  using  the  color  blue. 

Another  suggestion  offered  by  the  maintainers  was  to  use  color  to  identify  essential 
systems  based  upon  the  upcoming  sortie  type.  When  doing  an  assessment,  a  maintainer  could  tell 
the  model  that  the  next  sortie  for  the  aircraft  is  a  ferry  mission.  Based  on  the  sortie  type,  the 
model  could  dynamically  adjust  the  colors  used  to  identify  mission  essential  systems  for  the 
upcoming  sortie.  For  example,  all  of  the  essential  LRUs  for  the  ferry  sortie  in  the  radar  bay 
would  be  colored  in  red  and  non-essential  LRU  would  be  colored  in  blue. 

One  of  the  maintainers  mentioned  that  using  color  to  identify  electrical  systems  as  either 
AC  or  DC  would  be  very  helpful. 
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QUESTION:  Was  the  color  adequate  in  the  3D  graphics  just  used?  Why? 


All  maintainer  responded  that  the  color  in  the  3D  graphic  used  in  the  study  was  adequate. 
The  colors  in  the  model  as  closely  as  possible  to  resembling  the  actual  aircraft.  Two  of  the 
maintainers  mentioned  that  it  was  easy  on  the  eyes  and  that  it  made  it  easier  to  locate  the  damage 
areas.  One  of  the  maintainers  commented  that  it  was  not  necessary  to  try  to  match  the  colors  in 
the  model  exactly  to  the  aircraft  because  the  colors  are  not  necessarily  even  for  the  same  model 
and  block  number  of  aircraft.  The  maintainer  also  mentioned  that  it  would  be  more  important  to 
use  color  to  identify  different  structural  categories  than  to  try  to  match  the  exact  color  of  the 
aircraft. 

QUESTION:  Adequacy  of  color  in  the  3D  graphics. 


Rating 

Subiect  1 

Outstanding 

Subiect  2 

Satisfactory 

Subject  3 

Outstanding 

Subiect  4 

Satisfactory 

Subiect  5 

Outstanding 

Subiect  6 

Outstanding 

QUESTION:  Would  shading  he  helpful  in  the  use  of  3D  graphics?  Why? 

Four  of  the  maintainers  responded  that  shading  would  be  helpful  in  3D  graphics.  One  of 
the  maintainers  that  disagreed  mentioned  that  shading  would  be  helpful  as  long  as  shadows  were 
not  modeled.  This  maintainer  felt  that  modeling  shadows  would  cause  a  lot  of  confusion  when 
navigating  through  the  model.  After  explaining  to  this  maintainer  that,  the  question  about 
shading  was  only  intended  to  address  shading  and  not  shadowing,  the  maintainer  agreed  that  the 
use  of  shading  would  be  helpful.  Another  maintainer  felt  that  shading  is  beneficial  in  providing 
visual  cues  to  the  position  of  the  model. 

The  other  maintainer  who  commented  that  shading  would  not  be  helpful  mentioned  that  it 
hinders  the  assessor’s  ability  to  accurately  mark  the  damaged  areas. 
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QUESTION;  Was  the  shading  adequate  in  the  3D  graphics  just  used?  Why? 


Three  of  the  maintainers  responded  “yes”,  one  “no”  and  one  maintainer  did  not  answer  as 
a  result  of  not  noticing  the  use  of  shading  in  the  3D  graphic. 

QUESTION:  Adequacy  of  shading  in  the  3D  graphics. 

The  three  maintainers  felt  the  shading  was  an  important  element  in  making  the  3D 
graphics  more  realistic. 


Rating 

Subject  1 

Highly  Satisfactory 

Subject  2 

Satisfactory 

Subject  3 

Satisfactory 

Subject  4 

Satisfactory 

Subject  5 

Outstanding 

Subject  6 

Outstanding 

QUESTION;  Was  there  ever  a  time  in  the  task  just  performed  when  you  could  not  achieve 
a  proper  perspective?  When  specifically? 

Four  of  the  maintainers  responded  that  they  never  had  a  problem  achieving  a  proper 
perspective  and  two  of  the  maintainers  responded  with  “yes”.  The  two  maintainers  that 
responded  “yes”  mentioned  that  with  more  practice,  the  difficulty  of  obtaining  a  proper 
perspective  would  most  likely  are  eliminated. 


QUESTON:  Ease  in  obtaining  a  proper  perspective. 


Rating 

Subject  1 

Satisfactory 

Subiect  2 

Satisfactory 

Subject  3 

Satisfactory 

Subject  4 

Satisfactory 

Subject  5 

Satisfactory 

Subject  6 

Satisfactory 
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QUESTION:  Could  you  please  tell  us  if  these  requirements  were  met  in  the  task  just 
performed? 


Sub.  1 

Sub.  2 

Sub.  3 

Sub.  4 

Sub.  5 

Sub.  6 

Full  manipulation  of  the  3D  graphic 

Y 

Y 

Y 

Y 

Y 

Y 

Identification  of  components 

Y 

Y 

Y 

Y 

Y 

Y 

Caoabilitv  to  peel  away  layers  of  the  3D  graphic 

Y  1 

Y 

Y 

Y 

Y 

Y 

Shading  used  appropriately 

Y 

Y 

Y 

Y 

Y 

Y 

Use  of  undo  feature _ _ _ _ _ 

Y 

Y 

Y 

N 

Y 

Y 

Use  of  multiple  simultaneous  views 

NA 

NA 

NA 

Y 

Y 

N 

Use  of  measurement  units 

NA 

NA 

NA 

Y 

N 

N 

Use  of  wireframe  view 

NA 

NA 

NA 

Y 

Y 

N 

Use  of  orthogonal  views 

NA 

NA 

NA 

Y 

Y 

N 

OUESTON:  How  would  you  use  Cosmo/3D  Studio  Max  and  the  computer  just  used  to  aid 
in  the  efficiency  of  an  assessment? 

Three  of  the  maintainers  commented  that  by  utilizing  the  viewer  and  the  model,  an 
assessment  could  be  done  more  accurately  and  efficiently  than  it  is  currently  being 
accomplished.  However,  these  maintainers’  comments  are  predicated  on  the  fact  that  the  level  of 
detail  of  the  model  would  have  to  be  increased  in  order  for  the  system  to  be  used  in  practice.  One 
maintainer  commented  that  the  ability  to  mark  the  damage  on  the  model  while  doing  an 
assessment  would  be  very  helpful.  Once  the  damage  has  been  marked  on  the  model,  it  could  be 
sent  electronically  to  an  engineer,  possibly  at  another  base,  for  further  assessments  and 
collaboration.  Another  commented  that  having  all  of  the  information  readily  available  from  the 
3D  model  aided  in  the  efficiency  of  an  assessment,  as  currently  the  maintainer  may  have  to 
consult  multiple  TOs  to  do  an  assessment. 

The  remaining  maintainers  felt  that  more  detail  and  adequate  training  would  be  required 
for  the  model  to  be  helpful  in  aiding  in  the  efficiency  of  an  assessment. 

QUESTION:  Do  you  have  any  suggestions  as  to  the  use  and  manipulation  of  3D  graphics? 

All  but  one  of  the  maintainers  provided  suggestions  for  this  question.  Some  of  the 
responses  were  repetitive  statements  of  suggestions  provided  in  previous  questions.  One  of  the 
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maintainers  commented  that  the  system  needs  to  be  user  friendly  and  that  the  environment  and 
conditions  under  which  a  maintainer  operates  needs  to  be  taken  into  consideration.  Another 
maintainer  commented  that  the  more  detail  the  better  and  that  links  need  to  be  provided  to  the 
textual  information  contained  in  the  TOs. 

Another  suggestion  was  to  provide  a  small  image  of  the  aircraft  in  the  comer  of  the 
screen  that  tracked  the  user  point  of  view  in  the  main  model.  This  maintainer  commented  that 
there  were  instances  where  the  overall  orientation  was  lost  because  the  model  had  been  zoomed 
in  so  far.  The  user  also  commented  that  using  a  large  trackball,  similar  to  ones  found  in  arcade 
games,  would  make  navigating  through  the  model  much  easier. 

Finally,  two  of  the  maintainers  commented  that  the  model  was  great  and  the  program  was 
off  to  a  good  start. 

Preferred  Viewer  Features 

Most  of  the  features  available  to  the  maintainers  were  used.  As  previously  stated  two 
viewers,  Cosmo  and  3D  Studio  Max,  were  used  in  the  comparative  study.  Although  the  overall 
ease  of  manipulation  was  rated  higher  for  3D  Studio  Max  than  Cosmo,  a  selection  of  one  viewer 
over  the  other  was  not  the  goal  of  the  study.  To  the  contrary,  the  features  contained  in  each 
viewer  were  the  focus  of  concern.  The  viewers  contained  different  numbers  of  features.  The 
maintainers  that  evaluated  Cosmo  judged  the  seek,  rotate,  go,  remove/replace,  mark,  and 
information  features  as  being  essential  in  the  manipulation  of  3D  graphics.  The  maintainers  that 
used  3D  Studio  Max  stated  that  all  the  features  they  were  presented,  seek,  zoom,  rotate,  pan, 
undo/redo,  remove/replace  and  mark,  were  essential.  When  evaluating  features  across  viewers  it 
can  be  seen  that  the  seek,  rotate,  remove/replace  and  mark  features  were  very  essential  in 
performing  the  comparative  study  task. 

Because  only  a  subset  of  features  in  each  viewer  was  of  interest  in  this  study,  the 
maintainers  were  instructed  how  to  use  those  particular  features  and  ignore  the  others.  For  the 
majority  of  the  maintainers,  this  was  not  a  problem,  however  there  were  occasional  uses  of 
unapproved  features.  The  most  blatant  use  of  restricted  feature  was  found  in  Cosmo  when 
several  maintainers  used  the  straighten  feature.  Although  this  feature  was  readily  available,  it 


was  not  considered  crucial  for  the  manipulation  of  3D  graphics,  thus  not  evaluated.  However,  the 
maintainers  found  the  straighten  feature  to  be  extremely  useful.  In  future  studies,  a  wider  range 
of  features  should  be  selected  for  analysis. 

There  were  numerous  suggestions  for  additional  manipulation  features  that  were  not 
present  in  the  viewers.  One  maintainer  suggested  the  capability  to  positioning  the  model  by 
entered  coordinates  or  just  the  name  of  the  damage  area.  Several  maintainers  recommended  that 
having  the  ability  to  annotate  damage  on  the  model  would  be  very  beneficial.  This  statement 
exactly  duplicated  a  statement  made  in  the  CTA.  A  depiction  of  the  damage  would  only  help  in 
the  assessment  and  repair  processes. 

Conclusion 

The  maintainers  gave  a  favorable  response  to  the  use  of  a  laptop  computer  to  display  3D 
graphics.  They  suggested  that  they  would  have  preferred  to  use  a  trackball  or  joystick  rather  than 
the  external  mouse  used  in  the  study.  The  maintainers  were  only  given  two  choices  of  input 
devices,  either  the  stick  located  in  the  center  of  the  laptop  keyboard  or  the  external  mouse. 
Although  the  graphics’  realism  was  deemed  satisfactory,  there  still  was  a  call  for  additional 
detail.  Within  the  comparative  study’s  task,  all  the  maintainers  pointed  out  examples  of 
misplaced  or  missing  landmarks.  By  increasing  the  level  of  detail  in  the  graphics,  the  problem 
with  landmarks  should  be  eliminated.  When  asked  whether  maintainers  would  prefer  to  use 
either  the  computer  based  3D  graphics  or  multiple  fixed  perspective  paper  based  drawings,  the 
majority  responded  that  they  would  prefer  the  3D  graphics. 
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CONCLUSION 


It  has  become  apparent  throughout  the  course  of  this  study  that  there  is  a  lack  of  research 
regarding  the  use  of  3D  graphics  in  aircraft  maintenance.  This  work  begins  to  address  the  issues 
involved  with  the  potential  implementation  of  3D  graphics  within  TOs.  Many  factors,  such  as 
graphical  and  3D  manipulation  requirements,  need  to  be  clarified  before  the  benefit  of  this 
technology  can  be  fully  utilized.  This  study  has  identified  additional  research  efforts  that  can 
draw  from  laboratory  collaboration. 

Cognitive  Task  Analysis 

The  CTA  served  as  a  tool  to  uncover  several  inefficiencies  that  currently  exist  in  the  TO 
graphics  and  established  requirements  for  3D  graphics.  The  ultimate  goal  of  the  CTA  was  to 
construct  a  list  of  requirements  that  candidate  3D  file  formats  and  viewers  could  be  evaluated. 

The  main  inefficiencies  of  the  current  TO  graphics  revolved  around  limited  perspective 
of  a  graphic  to  describe  a  repair,  and  a  lack  of  detail  within  the  graphic.  As  described  by  the 
maintainers,  the  inability  to  change  the  graphic’s  perspective  made  some  otherwise  simple 
repairs  very  difficult.  Whether  it  is  an  inside-out  perspective  when  looking  at  it  from  outside  in, 
or  a  mirror  image  of  a  wing,  there  are  only  a  finite  number  of  views  that  paper-based  TO 
graphics  can  provide.  Along  with  only  a  finite  number  of  views,  paper-based  graphics  are  also 
limited  by  the  amount  of  information  they  can  contain.  Another  problem,  which  surfaced  during 
the  CTA,  is  the  apparent  lack  of  detail  in  the  current  TO  graphics.  The  maintainers  suggested  that 
the  level  of  detail  in  a  graphic  be  selectable  in  accordance  to  the  maintainer’s  background  and  the 
maintenance  task  being  performed.  For  instance,  an  electrician  and  a  sheet  metal  specialist  may 
want  to  see  different  levels  of  detail  when  looking  at  an  engine  bay.  The  ability  to  add  and 
remove  levels  of  detail  would  be  extremely  beneficial.  However,  at  any  level  an  electronic  3D 
graphic  should  be  more  inapt  to  convey  information  than  a  traditional  paper-based  TO.  The 
proposed  replacement  of  the  current  graphics  by  3D  graphics  is  in  an  effort  to  correct  these 
inefficiencies. 

During  the  CTA,  several  requirements  were  established.  These  included  but  were  not 
limited  to  the  corrections  of  the  inefficiencies  mentioned  above.  Labeling  of  components. 


removal  of  components,  and  having  full  manipulation  of  3D  graphics  were  the  primary 
requirements  gathered  from  the  CTA.  The  maintainers  expressed  interest  in  having  the  ability  to 
easily  locate  and  identify  components.  Once  having  located  the  components,  they  wanted  to  be 
able  to  manipulate  them  in  any  manner  they  felt  necessary  in  order  to  complete  the  repair.  These 
requirements  were  used  as  selection  criteria  for  the  3D  file  formats  and  viewers. 

Technical  Effort 

There  were  three  main  objectives  within  the  technical  effort,  1)  identification  of  “best 
value”  3D  file  formats  by  production  of  a  file  format  ranking  system,  2)  identification  of  “best 
value”  3D  viewers  by  production  of  a  3D  viewer  ranking  system,  and  3)  creation  of  a  3D  model 
and  accompanying  simulation  features. 

During  the  “best  value”  file  format  identification,  the  team  identified  and  ranked  industry 
and  DoD  3D  file  formats.  In  order  to  rank  the  file  formats,  a  set  of  weighted  criteria  were 
developed  and  used  to  give  each  of  the  file  formats  an  objective  numerical  ranking.  From 
ranking  the  file  formats,  it  was  found  that  VRML  and  SEDRIS  presented  the  best  value  choices 
for  representing  3D  graphics  in  electronic  TOs. 

During  the  “best  value”  3D  viewer  identifications,  the  team  investigated  several 
prominent  3D  simulation  areas:  3D  modeling  software,  CAD  packages,  and  virtual  reality 
software.  Similar  to  the  file  formats,  a  set  of  weighted  criteria  were  developed  and  used  to 
produce  an  objective  ranking.  From  ranking  the  software  viewers,  it  was  found  that  Cosmo 
Player  and  3D  Studio  Max  presented  the  best  two  simulation  environments  for  3D  viewing. 

In  both  the  production  of  the  3D  file  format  ranking  system  and  the  3D  viewer  ranking 
system,  a  set  of  objective  metrics  were  created.  These  metrics  and  ranking  systems,  when 
applied  to  3D  file  format  and  3D  viewer  products,  can  objectively  determine  the  best  products 
for  the  aircraft  maintenance  task.  Although  the  “best  of  the  best”  have  been  determined  at  this 
point  in  time,  the  3D  graphics  market  is  a  market  in  constant  flux,  rendering  the  best  products 
obsolete  in  a  matter  of  months.  These  ranking  systems  can  help  determine  the  best  products  in 
new  markets  and  are  intended  for  reuse  in  the  future. 
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Finally,  in  support  of  the  comparative  study,  a  3D  model  of  an  F-15  aircraft  with  a 
detailed  model  of  bay  3R  was  built.  This  model  was  used  in  conjunction  with  the  two  best  file 
formats  and  3D  viewers  to  create  two  models.  These  models  included  the  F-15  aircraft  model  as 
well  as  the  implementation  of  several  core  features  which  had  been  established  as  requirements 
from  the  CTA:  namely,  the  abilities  to  remove  and  replace  components,  fully  manipulate  the  3D 
graphics,  and  to  label  components. 


Comparative  Study 

The  comparative  study  was  conducted  to  determine  the  extent  to  which  3D  graphics 
would  be  useful  and  potentially  benefit  ABDR  maintenance.  Because  of  time  and  funding 
constants,  both  of  these  questions  were  only  addressed  though  a  questionnaire  rather  than  a 
performance  study.  All  but  one  of  the  six  maintainers  that  participated  in  the  study  stated  in  the 
questionnaire  that  3D  graphics  and  viewers  were  easy  to  use.  The  majority  of  the  maintainers 
responded  that  they  never  experienced  a  problem  achieving  a  proper  perspective  of  the  3D 
graphics.  Three  maintainers  commented  that  with  an  increased  level  of  detail  in  the  graphics  they 
could  conduct  assessments  more  accurately  and  efficient  than  what  is  currently  being 
accomplished. 

The  comparative  study  also  substantiated  the  3D  graphic  requirements  established  in  the 
CTA  along  with  gathering  additional  requirements.  These  additional  requirements  were 
prompted  by  the  illustration  of  the  3D  model  used  in  the  comparative  study  task.  Once  the 
maintainers  could  associate  the  functionality  of  an  actual  3D  model  with  a  maintenance  task, 
they  provided  comments  that  are  much  more  detailed. 

The  comparative  task  analysis,  technical  effort  and  the  comparative  study  all  combined  to 
establish  some  basic  requirements  for  3D  graphics  within  maintenance.  However,  these  alone  did 
not  spell  out  the  work  yet  to  be  done  before  this  technology  can  be  operational.  There  are 
numerous  areas  of  further  research. 

Areas  for  Further  Study 

A  list  of  3D  graphical  requirements  formed  from  the  CTA  was  presented  to  the 
maintainers  who  participated  in  the  comparative  study.  When  asked  if  the  3D  graphic  and  viewer 
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combination  used  in  the  comparative  task  met  these  requirements,  an  overwhelming  “yes” 
prevailed.  This  implies  that  the  use  of  3D  graphics  has  already  displayed  the  capability  of 
meeting  the  maintainer’s  current  requirements.  However,  there  were  several  suggestions  that 
would  increase  the  likelihood  of  acceptance.  Additional  user  friendliness  of  the  3D  viewer’s 
GUI,  an  increased  level  of  situation  awareness  of  their  repair  locations  within  the  aircraft,  and 
additional  graphical  detail  were  mentioned  as  essential  improvements. 

A  fundamental  issue  of  concern  is  the  level  of  graphical  detail  —  and  therefore  realism 
required.  Two  contributing  (and  unfortunately,  opposing)  factors  play  a  large  part;  frame  rate  and 
model  complexity.  Frame  rate  is  ultimately  determined  by  the  amount  of  model  complexity; 
simple  models  present  high  frame  rates  due  to  the  small  amount  of  computations  required  to 
render.  Complex  models  create  more  work,  and  allow  the  computer  to  update  the  screen  less 
often.  Both  high  frame  rates  and  complex  models  add  to  realism,  indicating  the  need  to  research 
the  balance  between  frame  rate  vs.  graphical  detail. 

Throughout  the  course  of  the  study,  several  questions  arose  concerning  the  viability  and 
efficiency  of  electronic  TOs  when  compared  to  the  current  paper-based  TOs,  even  more 
specifically  the  use  of  3D  graphics  within  electronic  TOs  compared  to  the  current  method  of 
performing  maintenance.  Although  there  is  a  trend  to  modernize  current  practices  in  the  Air 
Force  the  impact  of  such  modernization  is  not  always  fully  evaluated  prior  to  application. 
Questions  like  these  need  to  be  evaluated  and  tested  thoroughly  to  fully  understand  and  steer 
technology  to  the  best  possible  solution  for  all  involved. 

The  comparative  study  resolved  quite  a  few  practical  concerns  regarding  the  conditions  in 
which  maintainers  perform  repairs.  The  optimal  hardware  configuration  (e.g.,  input  device) 
needs  to  be  investigated.  Environmental  conditions  must  also  be  considered  for  the  usability  of 
electronic  TOs  to  be  fully  determined.  The  maintainers  that  participated  in  the  study  conveyed 
that  they  often  work  in  chemical  gear  to  perform  repairs.  Issues  such  as  clothing  and  the 
environment-conditions  need  to  be  evaluated. 

Several  media  have  been  studied  for  use  in  displaying  electronic  TOs  to  the  maintainers. 
This  study  used  a  laptop  computer  to  display  and  manipulate  3D  graphics.  Another  common 
approach  is  the  use  of  wearable  computing  systems  with  Head-Mounted  Displays  (HMD)  to 
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display  TOs.  There  are  many  application  questions  to  be  answered  for  both  of  these  media  in 
relation  to  3D  graphics.  A  study  to  evaluate  these  issues  would  guide  development. 

The  creation  of  the  F-15  model  in  the  investigation  revealed  the  need  to  create  or  find 
suitable  3D  models.  Although  the  model  can  be  created  from  scratch,  time  and  effort  could  be 
saved  from  finding  (and  converting)  existing  models  from  other  simulation  systems.  To  use  such 
a  model  there  appear  a  subset  of  problems  which  must  be  overcome  to  use  the  data,  namely: 

1)  Is  the  data  in  a  usable  file  format  or  can  it  be  converted  into  a  usable  file  format? 
Although  aircraft  can  be  designed  digitally,  they  may  have  been  designed  in  a 
proprietary  system  -  meaning  legal  considerations  will  have  to  be  taken  into  account. 
Additionally,  the  model  may  have  been  designed  in  a  system  which  uses  its  own  way 
of  storing  data  -  and  its  own  file  format,  meaning  file  format  conversions  will  have  to 
be  taken  into  account. 

2)  Is  the  data  too  complex  to  use?  The  amount  of  data  presented  in  the  file  format  may 
well  be  far  more  than  can  be  currently  handled  by  the  systems  needed.  If  so,  the 
process  of  distillation  from  the  original  set  of  data  to  the  needed  set  of  data  will  need 
to  be  investigated. 

Throughout  the  study  several  maintainers  stated  repetitively  that  the  utilizing  3D  graphics 
would  benefit  maintenance  training.  These  suggestions  were  given  a  lot  of  consideration 
especially  since  a  few  of  the  maintainers  participating  in  the  study  were  ABDR  instructors. 
Although  it  may  have  been  the  consensus  of  the  group  that  3D  graphics  could  quicken  the  pace 
of  training  in  a  more  efficient  manner,  the  substantiation  of  this  hypothesis  was  never  the  focus 
the  study.  A  much  more  detailed  project  would  need  to  be  designed  that  specifically  addresses 
this  question. 

Along  those  same  lines,  the  maintainers  were  asked  whether  they  thought  3D  graphics 
would  aid  in  ABDR  assessments,  after  they  completed  the  comparative  study  task.  Again, 
although  the  majority  of  the  maintainers  felt  that  3D  graphics  would  make  the  assessment 
process  more  efficient,  this  was  never  tested.  A  follow-on  study  that  could  compare  the  current 
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method  of  performing  assessments  with  a  new  method  that  utilized  3D  graphics  could  address 
this  question. 

Although  these  research  opportunities  only  represent  a  subset  of  the  potential  issues 
involved  with  implementing  3D  graphics  in  aircraft  maintenance  they  provide  a  base  from  which 
to  start.  With  any  implementation  of  new  technology,  there  are  many  challenges. 

It  became  very  evident  through  this  research  effort  sponsored  by  AFRL/HESR  that  the 
graphics  prevalent  with  in  the  current  paper  based  TOs  are  routinely  inadequate  and  sometimes 
even  misleading.  The  team  feels  that  this  effort  provided  for  a  great  first  step  in  helping  identify 
some  of  the  problems  and  issues  with  the  current  graphics  as  well  as  investigating  a  new 
alternative,  3D  graphics.  With  the  continual  advances  in  computer  hardware  and  software  arena, 
along  with  continuing  innovated  research  efforts  spearheaded  by  AFRL,  the  reality  of  providing 
fully  manipulatable  3D  graphics  for  use  in  electronic  TOs  is  attainable  now. 
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Glossary 

Backfaceng:  Method  of  reducing  polygons  drawn  by  the  simulation  by  calculating  the 
normal  to  the  polygon’s  surface. 

Browser:  Software  package  that  can  display  Hypertext  Markup  Language  (HTML) 
pages. 

Data  Visualization:  The  ability  to  display  a  set  of  data  in  a  graphical  form. 

Design  Prototyping  Software:  Software,  usually  related  to  CAD  packages,  which 
allows  objects  such  as  vehicles  or  aircraft  to  be  designed  in  a  simulated 
environment.  Objects  are  usually  designed  with  real-world  purposes  in  mind. 

Digital  Picture:  A  set  of  data  that  represents  a  2D  image. 

Frame  Rate:  Measurement  of  how  often  the  simulated  environment’s  screen  is  updated 
per  second. 

Modeling  Software:  Software  package  which  allows  objects  to  be  designed  in  a 
simulated  environment.  Objects  are  usually  designed  to  be  used  in  other 
simulated  environments. 

Movement  Paradigm:  A  method  for  movement  though  a  simulated  environment  that 
mimics  real  life  movement,  such  as  walking,  flying,  driving,  etc. 

Plug-IN:  a  software  package  that  works  in  conjunction  with  a  browser.  The  plug-in 

adds  some  functionality  to  the  browser  which  the  browser  did  not  originally  have. 
In  most  cases  in  this  investigation,  the  plug-in  added  the  ability  to  view  a  3D 
simulated  environment. 

Polygons:  Flat  planes  created  by  interconnecting  vertices. 

Rotation:  The  ability  to  rotate  an  object  or  eyepoint  in  a  simulated  environment. 

Stand-alone:  A  software  package  which  performs  a  task  with  no  additional  plug-ins 
needed  for  support. 

3D  File  Format:  A  protocol  for  a  set  of  data  which  represents  a  3D  object. 

3D  File  Format  Converter:  Software  that  can  import  a  3D  file  format,  understand  its 
contents,  and  export  those  contents  as  another  3D  file  format. 

3D  Viewer:  Software  that  allows  the  data  contained  in  a  3D  file  format  to  be  presented 
onscreen  in  a  3D  simulated  environment. 
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TimeSensor;  An  object  in  a  VRML  2.0  simulated  environment.  TimeSensors  have 
multiple  functionality;  the  function  used  in  this  investigation  was  to  display 
animation  frames  at  particular  intervals. 

TouchSENSOR:  An  object  in  a  VRML  2.0  simulated  environment,  used  to  determine 
mouse  click  input  from  the  user. 

Translation:  The  ability  to  move  an  object  or  eyepoint  in  a  simulated  environment 
from  one  position  to  another. 

Vertices:  Three  dimensional  (x,y,z)  points  in  a  3D  simulated  environment. 

Virtual  Reality  Software:  Software  package  that  displays  a  3D  simulated 
environment. 

Wireframe:  A  method  of  displaying  simulated  environments  in  which  all  objects  in  the 
simulation  appear  to  be  a  collection  of  lines  with  no  surfaces. 
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Appendix  A 

Background  Information  Questionnaire 
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Subject  Number: _ 

Scenario  Type:  ABDR 


Aircraft  Maintainer  Information  Questionnaire 


Name: _ 

Date; _ 

Title/Rank: _ 

AFSC: _ 

Organizational  Affiliation: 

AF  Base: _ 

Office  Phone: _ 


Type  of  Aircraft 
Maintained 

Approximate 

MonthsA'ears 

AF  Base 

Specialization 

Experience 

IHBH 

104 


Subject  Number: _ 

Scenario  Type:  ABDR 


Aircraft  Maintainer  Information  Questionnaire 

Name: _ _ _ 

Date: _ _ _ ^ - 

Title/Rank:  _ _ 

AFSC: _ _ _ 

Organizational  Affiliation:  _ _ 

AF  Base: _ _ _ _ _ 

Office  Phone: _ _ _ 


Type  of  Aircraft 
Maintained 


Approximate 

MonthsA'ears 


AF  Base 


ABDR 

Assessor  Qualified 


ABDR 

Specific  Qualified 


Appendix  B 

Maintenance  Task  Scenarios 
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Routine 

Foreign  Object  Damage  (FOD): 

“You  were  working  in  the  cockpit  and  you  dropped  a  screw,  please  walk  us  through  your 
process  of  finding  it.” 

Brake  Change: 

“Could  you  please  walk  us  through  a  routine  brake  change?” 

Inspections  by  Feel: 

“Has  there  ever  been  a  time  you  performed  an  inspection  or  repair  in  which  you  had  to  rely 
solely  on  feel  because  there  was  no  visual  accessibility?  What  modifications  did  you  have 
to  make  to  your  inspection  and  repair  processes  to  compensate  for  your  inability  to  view  the 
problem  area?” 

Perspective  Problem: 

“Have  you  ever  encountered  a  problem  with  the  perspective  view  of  a  T.O.  graphic 
(figure)?  If  so,  can  you  recall  a  particular  example  of  when  a  graphic’s  perspective 
complicated  your  repair?” 


ABDR 

Crew  Chief: 

“Could  you  please  walk  us  through  your  inspection  process  for  an  airplane  returning  from 
battle?  Are  there  requirements  that  must  be  performed  strictly  by  feel  because  there  is  no 
visual  accessibility?  If  so,  could  you  please  describe  them?” 

Electrician: 

“You  run  across  a  broken  or  damaged  wire  bundle,  please  walk  us  through  your  process  of 
identifying  what  needs  to  be  repaired.  Such  as-  identifying  where  the  wires  travel 
throughout  the  aircraft,  identifying  the  bulk  heads  they  connect  to,  and  locating  their 
termination  points.” 

Sheet  Metal: 

“When  you  assess  and  repair  a  damaged  aircraft  what  techniques  do  you  use  to  keep  from 
causing  more  damage?” 
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Appendix  C 
CTA  Concept  Maps 
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Scenario:  Foreign  Object  Damage  (FOD)  ROUTINE  MAINTENANCE  SCENARIO 

SUBJECT  1 
2/12/98 
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Scenario:  Fire  Loop  (Eiectricai  Component)  ROUTINE  MAINTENANCE  SCENARIO 

SUBJECT  2 
2/12/98 


Scenario:  Bird  Strike  ROUTINE  MAINTENANCE  SCENARIO 

SUBJECT  3 
2/12/98 
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Scenario:  Foreign  Object  Damage  (FOD)  ROUTINE  MAINTENANCE  SCENARIO 

SUBJECT  4 
2/12/98 
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referenced  to  identify  all  removable  items  in  the  cockpit. 


Scenario:  Brake  Change  ROUTINE  MAINTENANCE  SCENARIO 

SUBJECT  5 
2/13/98 
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Scenario:  C-141  Window  Post  Repair  ROUTINE  MAINTENANCE  SCENARIO 

SUBJECTS 

2/13/98 
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Scenario:  Crew  Chief  ABDR  MAINTENANCE  SCENARIO 

SUBJECT  1 
2/12/98 
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Scenario:  Electrical  ABDR  MAINTENANCE  SCENARIO 

SUBJECT  2 
2/12/98 
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Scenario:  Sheet  Metal  (Burn  Damage)  ABDR  MAINTENANCE  SCENARIO 

SUBJECT  3 
2/12/98 
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Scenario:  Sheet  Meta!  ABDR  MAINTENANCE  SCENARIO 

SUBJECT  4 
2/12/98 


Scenario:  Crew  Chief  ABDR  MAINTENANCE  SCENARIO 

SUBJECT  5 
2/13/98 
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Scenario:  Crew  Chief  ABDR  MAINTENANCE  SCENARIO 

SUBJECT  6 
2/13/98 
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leference  TO  to  identify  sub-system  damage 


Appendix  D 


Comparative  Study  Background  Information  Questionnaire 
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CSubject  Number; _ 

Viewer:  Cosmo  World  -  3D  Studio  Max 
Input  device:  Stick  Mouse 

Aircraft  Maintainer  information  Questionnaire 

Name:  _ ^ _ 

Date:  _ _ 

Title/Rank:  _ _ 

AFSC:  _ 

Organizational  Affiliation: _ 

AF  Base:  _ 

Office  Phone: _ 

Do  you  have  a  home  computer?  Yes _  No _ 

Computer  Experience:  _  months  /  years 


Do  you  have  experience  with  3D  graphics?  Yes _  No, 

If  yes,  how  many  years?  _ 


Do  you  have  experience  with  3D  graphic  viewers  (application  which  allows  you  to  view 
and  manipulate  3D  graphics?  Yes _  No _ 

If  yes,  which  ones? _ _ _ 
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Appendix  E 

Cosmo  and  3D  Studio  Questionnaires 
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Name: _  Date: - 

Subject: _ _  No.Viewer:  Cosmo  2.1 


_  Comparative  Study  Questionnaire 

1 .  Were  the  3D  graphics  easy  to  use? 

Why?  _ _ 


Yes 


No 


2.  Did  the  laptop  computer  hinder  your  ability  to  use  the  3D  graphics?  Yes  No 

Why?  _ _ _ — — - 


3.  Adequacy  of  the  screen  size  for  displaying  the  3D  graphics. 


5. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

Adequacy  of  the  input  device  for  manipulating  the  3D  graphics. 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

Use  of  the  laptop  computer  as  a  tool  to  manipulate  the  3D  graphics. 

Outstanding  Highly.  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

Were  the  3D  graphics  displayed  in  this  task  realistic  and  reliable?  Yes 

Why?  _ _ _ — - 


No 


7.  Realism  of  the  3D  graphics. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 
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8.  Were  there  any  misplaced  or  missing  landmarks  in  the  3D  graphics?  Yes 

If  there  were,  did  they  cause  problems  in  the  scenario?  Yes 

Why? _ _ _ 


9.  Could  you  please  provide  examples  of  additional  manipulation  features  that  would  be  helpful? 


10.  Please  mark  the  features  you  used  in  the  task  just  performed: 
Seek 
Zoom 
Rotate 
Pan 
Tilt 
Go 
Slide 

Undo/Redo 

Remove/Replace 

Mark 

Information 


1 1 .  Ease  in  using  the  seek  feature. 

1 - ^ ^ ^ - 1 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

12.  Please  rate  the  seek  feature. 

I - ^ - 1 

Essential  Useful  Useless 


13.  Ease  in  using  the  zoom  feature. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 
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14.  Please  rate  the  zoom  feature. 


Essential  Useful  Useless 

15.  Ease  in  using  the  rotate  feature.  _ ^ 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

16.  Please  rate  the  rotate  feature. 

Essential  Useful  Useless 

17.  Ease  in  using  the  pan  feature. 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

18.  Please  rate  the  pan  feature. 

Essential  Useful  Useless 

19.  Ease  in  using  the  tilt  feature. 

1 - ^ - H - ^ - 1 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

20.  Please  rate  the  tilt  feature. 

I - ^ - 1 

Essential  Useful  Useless 


se  in  using  the  gc 

1  feature. 

1 

1 

Outstanding 


Highly 

Satisfactory 


Satisfactory 


Marginal 


Unsatisfactory 
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22.  Please  rate  the  go  feature. 


Essential  Useful  Useless 


23.  Ease  in  using  the  slide  feature. 

1 - ^ ^ ^ - 1 

Outstcinding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

24.  Please  rate  the  slide  feature. 

Essential  Useful  Useless 

25.  Ease  in  using  the  undo/redo  feature. 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

26.  Please  rate  the  undo/redo  feature. 

I - ^ - 1 

Essential  Useful  Useless 


27.  Eai 


t^e  in  using  the  relmove/replace  feature 


Outstanding 


Highly 

Satisfactory 


Satisfactory 


Marginal  Unsatisfactory 


28.  Please  rate  the  remove/replace  feature. 


Essential  Useful  Useless 

29.  Ease  in  using  the  mark  feature. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 
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30.  Please  rate  the  mark  feature. 


Essential  Useful  Useless 


31 .  Ease  in  using  the  information  feature. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

32.  Please  rate  the  information  feature. 


Essential  Useful  Useless 

33.  Would  you  have  preferred  a  paper  engineering  drawing  as  opposed  to  a  3D 

graphic  like  the  one  you  just  manipulated  in  Cosmo?  Yes 

If  so  when?  _ _ _ _ 

34.  Could  you  complete  the  task  just  performed  with  multiple 
paper  based  drawings  or  multiple  pictures? 

If  Yes,  how  many  views  would  you  need? _ 

Would  you  need  views  with  components  removed? 

35.  Would  you  prefer  to  use  multiple  fixed  perspective  paper  based  drawings 
as  opposed  to  the  3D  graphic  you  just  used? 

Why? _ 

36.  Ease  in  manipulation  within  Cosmo. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 


Yes 

Yes 

Yes 


37.  Would  color  be  helpful  in  the  use  of  the  3D  graphics? 


Yes 


38.  Was  the  color  adequate  in  the  3D  graphics  just  used? 


Yes 


No 


Why? 


39.  Adequacy  of  color  in  the  3D  graphics. 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

40.  Would  shading  be  helpful  in  the  use  of  3D  graphics?  Yes  No 

Why? _ _ _ 


41 .  Was  the  shading  adequate  in  the  3D  graphics  just  used?  Yes  No 

Why?  _ _ _ 


42.  Adequacy  of  shading  in  the  3D  graphics. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 


43.  Was  there  ever  a  time  in  the  task  just  performed  when  you 

could  not  achieve  a  proper  perspective?  Yes 

When  specifically?^ _ _ _ — 


44.  Ease  in  obtaining  a  proper  perspective. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 


45.  Could  you  please  tell  us  if  these  requirements  were  met  in  the  task  just  performed? 


Full  manipulation  of  the  3D  graphic- 
identification  of  components  (text)- 
Capability  to  peel  away  layers  of  the  3D  graphic- 
Color  used  appropriately- 


Yes 

No 

Yes 

No 

Yes 

No 

Yes 

No 
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Shading  used  appropriately- 

Yes 

No 

Use  of  undo  feature- 

Yes 

No 

Use  of  multiple  simultaneous  views- 

Yes 

No 

Use  of  measurement  units- 

Yes 

No 

Use  of  wireframe  view- 

Yes 

No 

Use  of  orthogonal  views- 

Yes 

No 

46.  How  would  the  use  Cosmo,  3D  graphics,  and  the  computer  just  used  aid  in  the  efficiency  of  an 
assessment? 


47.  Do  you  have  any  suggestions  as  to  the  use  and  manipulation  of  3D  graphics? 
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Name:  _ 

Subject  No. 


Date:  _ 

Viewer:  3D  Studio 


_ Comparative  Study  Questionnaire _ 

48.  Were  the  3D  graphics  easy  to  use?  Yes 


49.  Did  the  laptop  computer  hinder  your  ability  to  use  the  3D  graphics?  Yes  No 

Why? _ _ _ _ _ _ 


50.  Adequacy  of  the  screen  size  for  displaying  the  3D  graphics. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

51 .  Adequacy  of  the  input  device  for  manipulating  the  3D  graphics. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

52.  Use  of  the  laptop  computer  as  a  tool  to  manipulate  the  3D  graphics. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

53.  Were  the  3D  graphics  displayed  in  this  task  realistic  and  reliable?  Yes  No 

Why? _ — - 


54.  Realism  of  the  3D  graphics. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 
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55.  Were  there  any  misplaced  or  missing  landmarks  in  the  3D  graphics?  Yes  No 

If  there  were,  did  they  cause  problems  in  the  scenario?  Yes  No 

Why? _ _ _ _ _ 


56.  Could  you  please  provide  examples  of  additional  manipulation  features  that  would  be  helpful? 


57.  Please  mark  the  features  you  used  in  the  task  just  performed: 
Seek 
Zoom 

Zoom  Extents 

Rotate 

Pan 

Undo/Redo 

Remove/Replace 

Mark 

Information 


58.  Ease  in  using  the  seek  feature. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 


59.  Please  rate  the  seek  feature. 

Essential  Useful  Useless 

60.  Ease  in  using  the  zoom  feature. 

I - ^ ^ ^ - 1 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

61 .  Please  rate  the  zoom  feature. 

Essential  Useful  Useless 
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62.  Ease  in  using  the  zoom  extents  feature. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

63.  Please  rate  the  zoom  extents  feature. 

I - ^ - 1 

Essential  Useful  Useless 

64.  Ease  in  using  the  rotate  feature. 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

65.  Please  rate  the  rotate  feature. 

Essential  Useful  Useless 

66.  Ease  in  using  the  pan  feature. 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

67.  Please  rate  the  pan  feature. 

I - ^ - 1 

Essential  Useful  Useless 

68.  Ease  in  using  the  undo/redo  feature. 

I - ^ ^ ^ - 1 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

69.  Please  ratfe  the  undo/redo  feature. 

I - ^ - 1 

Essential  Useful  Useless 
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Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 


71.  Please  rate  the  remove/replace  feature. 


Essential  Useful  Useless 

72.  Ease  in  using  the  mark  feature. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

73.  Please  rate  the  mark  feature. 


Essential  Useful  Useless 


74.  Ease  in  using  the  information  feature 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

75.  Please  rate  the  information  feature. 


Essential  Useful  Useless 

76.  Would  you  have  preferred  a  paper  engineering  drawing  as  opposed  to  a  3D 

graphic  like  the  one  you  just  manipulated  in  Cosmo?  Yes 

If  so  when? _ 

77.  Could  you  complete  the  task  just  performed  with  multiple 
paper  based  drawings  or  multiple  pictures? 

If  Yes,  how  many  views  would  you  need? _ 

Would  you  need  views  with  components  removed? 


Yes  No 


Yes  No 
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78.  Would  you  prefer  to  use  multiple  fixed  perspective  paper  based  drawings 
as  opposed  to  the  3D  graphic  you  just  used? 

Why? _ _ _ 


79,  Ease  in  manipulation  within  3D  Studio  Max. 

1 - H — ^ ^ - F - 1 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

80.  Would  color  be  helpful  in  the  use  of  the  3D  graphics?  Yes 

Why?_ _ _ _ _ _ 


81 .  Was  the  color  adequate  in  the  3D  graphics  just  used? 
Why? _ _ _ 


Yes  No 


82.  Adequacy  of  color  in  the  3D  graphics. 

- ^ ^ H 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

83.  Would  shading  be  helpful  in  the  use  of  3D  graphics?  Yes 

Why? _ _ _ _ _ 


No 


84.  Was  the  shading  adequate  in  the  3D  graphics  just  used? 
Why? _ _ _ 


Yes  No 


85.  Adequacy  of  shading  in  the  3D  graphics. 


Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 
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86.  Was  there  ever  a  time  in  the  task  just  performed  when  you 

could  not  achieve  a  proper  perspective?  Yes  No 

When  specifically? _ _ _ 


87.  Ease  in  obtaining  a  proper  perspective. 

Outstanding  Highly  Satisfactory  Marginal  Unsatisfactory 

Satisfactory 

88.  Could  you  please  tell  us  if  these  requirements  were  met  in  the  task  just  performed? 


Full  manipulation  of  the  3D  graphic- 

Yes 

No 

identification  of  components  (text)- 

Yes 

No 

Capability  to  peel  away  layers  of  the  3D  graphic- 

Yes 

No 

Color  used  appropriately- 

Yes 

No 

shading  used  appropriately- 

Yes 

No 

Use  of  undo  featu re¬ 

Yes 

No 

use  of  multiple  simultaneous  views- 

Yes 

No 

Use  of  measurement  units- 

Yes 

No 

Use  of  wireframe  view- 

Yes 

No 

Use  of  orthogonal  views- 

Yes 

No 

89.  How  would  the  use  3D  Studio  Max,  3D  graphics,  and  the  computer  just  used  aid  in  the  efficiency  of 
an  assessment? 


90.  Do  you  have  any  suggestions  as  to  the  use  and  manipulation  of  3D  graphics? 
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Appendix  F 

Comparative  Study  Protocol 
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Comparative  Study  Protocol 
Robins  Air  Force  Base 
18-21  May _ 


The  Comparative  Study  consisted  of  the  following  five  sections,  which  are  explained 
below; 

1 .  Training 

2.  Scenario  Description 

3.  Task 

4.  Data  Collection 

5.  Questionnaire 

No  fewer  than  six  maintainers  participated  in  the  Comparative  Study.  Each  maintainer 
received  only  one  3D  viewer  to  evaluate.  The  Comparative  Study  lasted  two  hours  per 
maintainer. 


1.  Training  (30  minutes) 

Computer 

-  The  maintainers  were  given  time  to  become  familiar  with  the  laptop  computer. 
Input  Device 

-  The  maintainers  were  asked  to  select  one  of  the  following  input  devices: 

a.  External  mouse 

b.  Stick  located  in  middle  of  laptop  keyboard 

If  necessary,  the  maintainers  were  given  time  to  become  familiar  with  using  a 
mouse  by  playing  a  game  of  Solitaire. 

Viewers 

-  The  maintainers  were  instructed  how  to  use  all  the  features  required  to  complete 
the  task  (i.e.,  zoom,  pan,  rotate,  remove,  replace,  mark,  and  walk)  for  their 
particular  viewer. 

-  The  maintainers  were  asked  to  perform  the  following  simple  tasks  to  test  their 
ability  to  use  the  required  features. 

a.  Box  sitting  on  something  (start  with  a  far  away  view  and  have  them  zoom 
and  pan) 

b.  Box  with  something  written  on  each  side  (allows  them  to  rotate  and  walk 
the  box  to  read  each  of  the  sides) 

c.  Box  with  a  circle  inside  (allows  them  to  click  on  the  box  and  then  have  it 
disappear  and  the  circle  inside  appear,  remove  and  replace) 

■  d.  Box  with  a  bulls-eye  on  two  sides  (allows  them  to  place  a  mark  (cursor) 
inside  the  bulls-eye) 

-  Training  was  completed  when  the  maintainers  could  perform  all  the  above 
actions  without  assistance. 
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2.  Scenario  Description  (include  time  for  training) 

-  The  maintainers  were  given  the  following  scenario: 

“During  a  pilot  de-brief  the  pilot  explains  that  shortly  after  sustaining  a  hit  the 
radar  became  locked.  You  are  instructed  to  identify  and  repair  the  problem.” 


3.  Task  (30  minutes) 

The  maintainers  proceeded  to  the  plane  with  the  laptop  after  the  scenario  had  been 
explained.  Once  at  the  plane,  they  were  told  to  experiment  with  the  view  and  graphics  so 
that  they  feel  comfortable. 

Task  parameter:  The  initial  view  of  the  plane  was  at  an  angle  that  required  the 
maintainers  to  rotate  and  zoom  to  view  the  necessary  components. 

Completion  of  the  task  was  signified  by  the  following: 

a.  Placing  a  mark  on  the  3D  graphic  to  identify  all  the  damaged  components. 
Maintainers  should  place  marks  on  the  graphic  wherever  marks  appeared  on 
the  plane.  The  print  screen  was  used  to  capture  the  marker  on  the  3D  graphic. 

b.  Verbally  identify  the  components  that  were  damaged. 


4.  Data  Collection  (gathered  during  task) 

Response  times  identified  when  and  how  often  the  viewer’s  features  were  recorded  (i.e., 
start  task,  open  door  3R,  zoom  components,  pan  components,  rotate  components,  remove 
components,  replace  components,  walk  around  components,  mark  damage  and  complete 
task) 

5.  Questionnaire  (60  minutes) 

Please  refer  to  Appendix  D,  “Comparative  Study  Background  Information 
Questionnaire.” 


139 


